Tensorflow Serving¶
from kashgari.tasks.classification import BiGRU_Model
from kashgari.corpus import SMP2018ECDTCorpus
from kashgari import utils
train_x, train_y = SMP2018ECDTCorpus.load_data()
model = BiGRU_Model()
model.fit(train_x, train_y)
# Save model
utils.convert_to_saved_model(model,
model_path="saved_model/bgru",
version=1)
Then run tensorflow-serving.
docker run -t --rm -p 8501:8501 -v "<path_to>/saved_model:/models/" -e MODEL_NAME=bgru tensorflow/serving
Load processor from model, then predict with serving.
We need to check model input keys first.
import requests
res = requests.get("http://localhost:8501/v1/models/bgru/metadata")
inputs = res.json()['metadata']['signature_def']['signature_def']['serving_default']['inputs']
input_sample_keys = list(inputs.keys())
print(input_sample_keys)
# ['Input-Token', 'Input-Segment']
If we have only one input key, aka we are not using BERT like embedding, we need to pass json in this format to predict endpoint.
{
"instances": [
[2, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[2, 9, 41, 459, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0]
]
}
Here is the code.
import requests
import numpy as np
from kashgari.processors import load_processors_from_model
text_processor, label_processor = load_processors_from_model('/Users/brikerman/Desktop/tf-serving/1603683152')
samples = [
['hello', 'world'],
['你', '好', '世', '界']
]
tensor = text_processor.transform(samples)
instances = [i.tolist() for i in tensor]
# predict
r = requests.post("http://localhost:8501/v1/models/bgru:predict", json={"instances": instances})
predictions = r.json()['predictions']
# Convert result back to labels
labels = label_processor.inverse_transform(np.array(predictions).argmax(-1))
print(labels)
If we are using Bert, then we need to handle multi input fields,
for example we get this two keys ['Input-Token', 'Input-Segment']
from metadata endpoint.
Then we need to pass a json in this format.
[
{
"Input-Token": [2, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
"Input-Segment": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
},
{
"Input-Token": [2, 9, 41, 459, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0],
"Input-Segment": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
}
]
Here is the code.
import requests
import numpy as np
from kashgari.processors import load_processors_from_model
text_processor, label_processor = load_processors_from_model('/Users/brikerman/Desktop/tf-serving/1603683152')
samples = [
['hello', 'world'],
['你', '好', '世', '界']
]
tensor = text_processor.transform(samples)
instances = [{
"Input-Token": i.tolist(),
"Input-Segment": np.zeros(i.shape).tolist()
} for i in tensor]
# predict
r = requests.post("http://localhost:8501/v1/models/bgru:predict", json={"instances": instances})
predictions = r.json()['predictions']
# Convert result back to labels
labels = label_processor.inverse_transform(np.array(predictions).argmax(-1))
print(labels)