The filesystem - a quick & dirty way to store temp data in your API
I have an API service for a Google Sheets add-on that I built, and I wanted to temporarily store some request metadata that I started collecting in the client. How the add-on works. The API generates the forecast data. The service was built with Django Ninja, and it's not connected to a database. I thought about setting one up for this feature. But I couldn't be bothered dealing with the configuration and the migrations. I wasn't sure how long I'd be collecting this data for. I then considered dumping the metadata in the logs. The problem with that approach is that it's a hassle to aggregate all the relevant messages. I was planning to move the information to another place to do some analysis. I briefly reflected on a global variable as well, but I was worried about the memory management. And then, at last, I got to the most robust solution in the galaxy - the filesystem. I would store each piece of request metadata in a separate file, so as to avoid issues with concurrent writes. And, I would expose an endpoint in the API that combines all those files to show me all of the data. Voilà the code: def request(): # at the end, before I return the response entry = { "timestamp": datetime.now().isoformat(), "request_id": request_id, "metadata": metadata, } with open(f"{REQUESTS_DIR}/{request_id}.json", "w") as f: json.dump(entry, f) return response # decorator to authenticate with API key header def get_logs(): requests = [] for filename in glob.glob(f"{REQUESTS_DIR}/*.json"): with open(filename, "r") as f: try: data = json.load(f) # Add filename as id for reference data["id"] = os.path.basename(filename).replace(".json", "") requests.append(data) except json.JSONDecodeError: pass # Sort by timestamp requests.sort(key=lambda x: x["timestamp"]) return requests With that deployed, I can now fetch all of the data in my API's OpenAPI docs. Django Ninja generates this page automatically for me. Lovely jubbly! The OpenAPI endpoints The OpenAPI response for the endpoint I just created I started this blog because I wanted to have conversations on topics that I find interesting. So, feel welcome to reach out to me, even if to challenge my perspective. I'm on LinkedIn; please add a message when you connect.
I have an API service for a Google Sheets add-on that I built, and I wanted
to temporarily store some request metadata that I started collecting in the client.
How the add-on works. The API generates the forecast data.
The service was built with Django Ninja, and it's not connected to a database. I thought about setting one up for this feature. But I couldn't be bothered dealing with the configuration and the migrations. I wasn't sure how long I'd be collecting this data for.
I then considered dumping the metadata in the logs. The problem with that approach is that it's a hassle to aggregate all the relevant messages. I was planning to move the information to another place to do some analysis.
I briefly reflected on a global variable as well, but I was worried about the memory management.
And then, at last, I got to the most robust solution in the galaxy - the filesystem. I would store each piece of request metadata in a separate file, so as to avoid issues with concurrent writes. And, I would expose
an endpoint in the API that combines all those files to show me all of the data.
Voilà the code:
def request():
# at the end, before I return the response
entry = {
"timestamp": datetime.now().isoformat(),
"request_id": request_id,
"metadata": metadata,
}
with open(f"{REQUESTS_DIR}/{request_id}.json", "w") as f:
json.dump(entry, f)
return response
# decorator to authenticate with API key header
def get_logs():
requests = []
for filename in glob.glob(f"{REQUESTS_DIR}/*.json"):
with open(filename, "r") as f:
try:
data = json.load(f)
# Add filename as id for reference
data["id"] = os.path.basename(filename).replace(".json", "")
requests.append(data)
except json.JSONDecodeError:
pass
# Sort by timestamp
requests.sort(key=lambda x: x["timestamp"])
return requests
With that deployed, I can now fetch all of the data in my API's OpenAPI docs. Django Ninja generates this page automatically for me. Lovely jubbly!
The OpenAPI endpoints
The OpenAPI response for the endpoint I just created
I started this blog because I wanted to have conversations on topics that I find interesting. So, feel welcome to reach out to me, even if to challenge my perspective. I'm on LinkedIn; please add a message when you connect.
