A hard rain's a-gonna fall: decoding JSON in Rust

From The Secrets of Rust: Tools

Photo by Malcolm Lightbody on Unsplash

Heard ten thousand whisperin’ and nobody listenin’.
—Bob Dylan, “A Hard Rain’s a-Gonna Fall”

JSON is the worst data format, apart from all the others, but here we are. This is the life we chose, and if we’re writing Rust programs to talk to remote APIs such as Weatherstack, as we did in Here comes the sun, we’ll have to be able to cope with them sending us JSON data.

We’ll also need a way to turn that JSON into good honest Rust data, so we can compute stuff about it. Let’s see how to do that, as we continue to hack on our embarrassingly basic weather client program.

Extracting weather data from the response

In Elephants for breakfast we pondered how to test a function such as get_weather when we can’t know in advance what weather conditions it’ll return. The correct response is literally up in the air.

So we deftly flipped the problem—judo chop!—and decided instead to write two functions that are testable: request, which constructs the HTTP request to be sent to Weatherstack, and deserialize, which unpacks the answer.

We’ve already written request, or rather you did—great job on that, by the way! Let’s turn to deserialize now. Here’s how we plan to call it as part of the get_weather function:

let weather = deserialize(&resp.text()?)?;

At this point we’ve already made the API request, and now resp contains the response, consisting of weather data in a specific JSON format.

We’re saying here that if we pass the body of that response as a &str to deserialize, we should get back a Weather struct representing the weather conditions that were encoded in the JSON.

That sounds like something we can test, so it’s over to you again to figure out how.

GOAL: Write a test for deserialize along these lines.

---

HINT: We have the real JSON data we saved earlier when making our exploratory request to Weatherstack (if not, make it again). That’ll be perfect test data for deserialize: we already know exactly what weather conditions it encodes, so all we need to do is check the result against the corresponding Weather struct.

---

SOLUTION: Here’s my attempt:

#[test]
fn deserialize_extracts_correct_weather_from_json() {
    let json = fs::read_to_string("tests/data/ws.json").unwrap();
    let weather = deserialize(&json).unwrap();
    assert_eq!(
        weather,
        Weather {
            temperature: 11.2,
            summary: "Sunny".into(),
        },
        "wrong weather"
    );
}

(Listing weather_2)

There’s no need to test extracting any of the other stuff in the JSON; we don’t use it, so extracting it would be a waste of time, and testing that extraction even more so.

Deserializing the JSON

That was easy, I think you’ll agree, so let’s turn to the actual extraction. How are we going to turn a &str into a Weather?

You know if you’ve read my moderately bestselling book The Secrets of Rust: Tools that we can use the serde library to serialize and deserialize Rust structs to JSON by using the derive attribute. For example:

#[derive(Serialize, Deserialize)]
pub struct Memos {
    path: PathBuf,
    pub inner: Vec<Memo>,
}

By deriving the Serialize trait, we asked serde to autogenerate code for turning a Memos struct into, effectively, a &str, and Deserialize, naturally enough, does the reverse.

So could we do the same kind of thing here? Could we define some Rust struct that mirrors the schema of the JSON data returned by Weatherstack, and derive Deserialize on it?

A surfeit of structs

Yes, we could do it that way, but it turns out to be rather laborious, because the API’s schema consists of nested structures. We’d have to define structs for each level of the JSON we’re interested in. For example, at the top level, we only want current:

{
  "current": {
    ...
  },
  ...
}

So we have to start by defining a struct that represents the entire response, with a single field for current (we can ignore all the others):

struct WSResponse {
    current: WSCurrent,
}

And, of course, we now need another struct definition to represent what’s inside current:

struct WSCurrent {
    temperature: f64,
    weather_descriptions: Vec<String>,
}

It’s already getting annoying, and you can imagine there would be many more of these structs if we had to deal with further levels of nesting in the API data. The worst part is that we don’t even want these structs! There’s no function in our program that needs to take or return a WSCurrent, for example: we already have our own struct Weather that contains exactly and only the data we want.

Using JSON Pointer

Surely this isn’t the right way to use Rust’s type system. What we’d prefer is a way to look up the data we want directly in the JSON, and then transfer it to our Weather struct, without going via a bunch of useless paperwork.

Luckily, serde_json provides a way to do this, using a syntax called “JSON Pointer”. First, we deserialize the data to the all-purpose type json::Value:

use serde_json::Value;

let val: Value = serde_json::from_str(json)?;

Assuming that we successfully get val, then its contents represent the whole JSON object contained in the response. We can reach in and grab some specific part of it using a path-like notation:

let temperature = val.pointer("/current/temperature")

Much more direct than using a bunch of intermediate structs. Of course, there might not be a value at that path, so it makes sense that pointer returns an Option, doesn’t it? (Read more about this in my tutorial on Results and Options in Rust.)

If this path didn’t exist in the JSON, the result would be None, straightforwardly. But we can be pretty confident that the server’s response will include this data, so the result will be Some, and it will contain another json::Value representing whatever was found at the given path.

Implementing deserialize

So let’s try to write deserialize now using this “pointer” approach:

fn deserialize(json: &str) -> Result<Weather> {
    let val: Value = serde_json::from_str(json)?;
    let temperature = val
        .pointer("/current/temperature")
        .and_then(Value::as_f64)
        .with_context(|| format!("bad response: {val}"))?;
    let summary = val
        .pointer("/current/weather_descriptions/0")
        .and_then(Value::as_str)
        .with_context(|| format!("bad response: {val}"))?
        .to_string();
    Ok(Weather {
        temperature,
        summary,
    })
}

(Listing weather_2)

We’ve added a little extra paperwork here in case the lookups fail, using with_context to return a suitable error message along with the problematic JSON data.

Assuming the lookups succeed, though, we need to turn the resulting json::Values into real Rust types:

.and_then(Value::as_f64)

and_then is a useful little tool whenever we’re dealing with Options like this. If the option is None, it just does nothing. But if it’s Some, then it extracts the value and applies the given function to it. In this case, that’s as_f64 for the temperature, which parses the value as a floating-point number, and as_str for the summary.

Having extracted, checked, and converted our temperature and summary values, then, we write them into our Weather struct using the field init shorthand, and return it.

Taking it for a trial run

This passes the test, which is encouraging, so we now have the two magic functions we need to write get_weather.

Here it is again:

pub fn get_weather(
    location: &str,
    api_key: &str,
) -> Result<Weather> {
    let resp = request(location, api_key).send()?;
    let weather = deserialize(&resp.text()?)?;
    Ok(weather)
}

(Listing weather_2)

I mean, if you cross off the parts we’ve already tested (request and deserialize), there’s not much left, is there? The only substantive thing we don’t test is send(), and that’s not our code—we can assume reqwest itself works, or someone would have noticed by now.

Let’s run the updated program for real and see what happens.

cargo run

Error: bad response: {"error":{"code":601,"info":"Please specify a
valid location identifier using the query parameter.","type":
"missing_query"},"success":false}

Oops. That’s on me; I didn’t give a location on the command line to query the weather for. But that’s exactly the sort of mistake that any user might make, so we’d better catch it and provide a slightly nicer error message:

if location.is_empty() {
    bail!("Usage: weather <LOCATION>");
}

Let’s try again, this time with a location:

cargo run London,UK

Weather { temperature: 12.0, summary: "Partly cloudy" }

A nicer Display

Fine. All our magic is working perfectly. That’s reassuring, so let’s take this opportunity to tighten up a few bolts and caulk a few seams. We’ll define a real implementation of Display, so that the output doesn’t look so nerdy:

impl Display for Weather {
    fn fmt(
        &self,
        f: &mut std::fmt::Formatter<'_>,
    ) -> std::fmt::Result {
        write!(f, "{} {:.1}ºC", self.summary, self.temperature)
    }
}

(Listing weather_2)

Nothing new here except this format parameter for the temperature:

{:.1}ºC

The .1 means “print to one decimal place”, rounding if necessary. Without this, a temperature of 12.0 would print as just “12”, which seems a shame. We worked to get that extra decimal place: let’s hang on to it!

Here’s what it looks like with this change:

Partly cloudy 12.0ºC

Mixing arguments and environment variables

And, since we’re going to the trouble of validating our arguments, let’s hand that over to clap and derive a suitable Args parser:

#[derive(Parser)]
/// Shows the current weather for a given location.
struct Args {
    #[arg(required = true)]
    /// Example: "London,UK"
    location: Vec<String>,
}

You might well think that if we’re deploying the awesome power of clap now, couldn’t we also use it to take the API key as a command-line option? That would be a nice enhancement, but currently we’re looking for it in an environment variable:

let api_key = env::var("WEATHERSTACK_API_KEY")?;

Ideally, we’d have clap get this from a flag if it’s provided that way, and if not, look for it in the environment variable. And it turns out we can do exactly that, if we opt in to the env feature:

cargo add clap -F derive,env

Now we can write:

struct Args {
    #[arg(short, long, env = "WEATHERSTACK_API_KEY", required = true)]
    /// Weatherstack API key
    api_key: String,
    ...

What we’re saying is, if the user provides the --api-key flag, use that value, and if they don’t, fall back to looking for it in the environment variable. If it’s not there either, report an error, because this is a required flag.

By the way, while it’s conventional for Rust field names to be styled in so-called “snake case” (api_key), it’s also conventional for command-line arguments to be styled in so-called “kebab case” (--api-key), so clap makes this transformation for us automatically. That’s the wonderful thing about conventions, of course: there are so many of them.

Very well, then. We’ve built a decent library crate that actually gets weather, and it has some tests. We can be pretty confident that both request and deserialize are implemented properly. Since we can’t test the way they’re glued together in get_weather, I guess we’ll just have to hope it’s correct, right?

It’s correct, right?

If you don’t think “hope” is a strategy, tune in next time to see whether we can’t do a little better.