Skip to main content

Inference API

After deploying your cluster and creating an API key, you can now start to make an inference to your inference servers!

Example with python (httpx)​

Here's an example of a cURL command to help you test the request:

import httpx
client = httpx.Client(http2=True, timeout=httpx.Timeout(30.0))
response = client.post(endpoint_url, 
	headers={
   		'content-type': 'application/json',
		  'x-api-key': api_key
	}, json = {
		  'project_id': project_id,
		  'grpc_data': {'array':data_values.tolist()}
	})
output = response.json()

Explanation​

Inference Request Summary

Information Needed​

From the deployment tab in your project, there are a few necessary information you will require to make an inference request.

  • Endpoint URL
  • Project ID

Sending a Request​

To make an inference rquest, follow the steps below to make a GET request to the /api/inference/new endpoint.

Request Headers​

{
  "x-api-key": "<api_key>"
}

Request Body​

{
  "project_id": "<string>",
  "grpc_data": {
    "data": [<array of data>],
	"dtype": <dtype>
  }
}

Dtype Options​

Select the option based on the Model Configuration During Project Creation

  • "bool"
  • "uint8"
  • "uint16"
  • "uint32"
  • "uint64"
  • "int8"
  • "int16"
  • "int32"
  • "int64"
  • "float16"
  • "float32"
  • "float64"
  • "String"
  • "bf16"

HTTP Version​

HTTP/2 is recommended for use with inference for better