pyproc — Call Python from Go without CGO or microservices

I built pyproc to let Go services call Python like a local function — no CGO and no separate microservice. It runs a pool of Python worker processes and talks over Unix Domain Sockets on the same host/pod, so you get low overhead, process isolation, and parallelism beyond the GIL.

Why this exists

• Keep your Go service, reuse Python/NumPy/pandas/PyTorch/scikit-learn.
• Avoid network hops, service discovery, and ops burden of a separate Python service.

Quick try (~5 minutes)

Go (app):

go get github.com/YuminosukeSato/pyproc@latest  

Python (worker):

pip install pyproc-worker  

Minimal worker (Python):

from pyproc_worker import expose, run_worker  
@expose  
def predict(req):  
    return {"result": req["value"] * 2}  
if __name__ == "__main__":  
    run_worker()  

Call from Go:

import (  
  "context"  
  "fmt"  
  "github.com/YuminosukeSato/pyproc/pkg/pyproc"  
)  
func main() {  
  pool, _ := pyproc.NewPool(pyproc.PoolOptions{  
    Config:       pyproc.PoolConfig{Workers: 4, MaxInFlight: 10},  
    WorkerConfig: pyproc.WorkerConfig{SocketPath: "/tmp/pyproc.sock", PythonExec: "python3", WorkerScript: "worker.py"},  
  }, nil)  
  _ = pool.Start(context.Background())  
  defer pool.Shutdown(context.Background())  
  var out map[string]any  
  _ = pool.Call(context.Background(), "predict", map[string]any{"value": 42}, &out)  
  fmt.Println(out["result"]) // 84  
}  

Scope / limits

• Same-host/pod only (UDS). Linux/macOS supported; Windows named pipes not yet.
• Best for request/response payloads ≲ ~100 KB JSON; GPU orchestration and cross-host serving are out of scope.

Benchmarks (indicative)

• Local M1, simple JSON: ~45µs p50 and ~200k req/s with 8 workers. Your numbers will vary.

What’s included

• Pure Go client (no CGO), Python worker lib, pool, health checks, graceful restarts, and examples.

Docs & code

• README, design/ops/security docs, pkg.go.dev: https://github.com/YuminosukeSato/pyproc

License

• Apache-2.0. Current release: v0.2.x.

Feedback welcome

• API ergonomics, failure modes under load, and priorities for codecs/transports (e.g., Arrow IPC, gRPC-over-UDS).