back

by apwheele·3y ago·view on hn ↗
For a random python/pandas trick, I have come across web-api's that cannot be directly read into pandas using the URL (I imagine folks on here can comment better the difference in web serving tech), but you can read in the IO object and pass that to pandas. Blog post, https://andrewpwheeler.com/2022/11/02/using-io-objects-in-py..., but can just put simple example in comment:

    ####
    import pandas as pd
    from io import StringIO
    import requests
    url = ('https://data.townofcary.org/explore/dataset/cpd-incidents/download/'
           '?format=csv&timezone=America/New_York&lang=en&use_labels_for_header=true'
           '&csv_separator=%2C')
    res = requests.get(url)
    df = pd.read_csv(StringIO(res.text))
    ####
1 comments
For what it's worth, if the requests module works fine you could probably set "stream=True" in the request and read the `res.raw.data` file object directly. That way you avoid loading the data into memory first [1]. You'll probably want to set `res.raw.decode_content = True` to ensure you get the raw bytes, and not some zipped stream.

[1]: https://stackoverflow.com/questions/16923898/how-to-get-the-...