• for

• map

• doseq

• run!

In my experience, while the official documentation exists, it's often unclear when to use each construct. This guide breaks down the differences with practical examples, so you can make the right choice every time.

TLDR: Always use run! and doseq for side-effects, and map and for for lazy data transformations

1. map - Transform Collections

Purpose: Transform each element of a collection into something else.

Returns: A lazy sequence of transformed values.

When to use:

• You want to convert one collection into another

• You're applying a transformation function

• You need the result for further processing

Characteristics:

• Lazy - only computes values when needed

• Returns a sequence - the transformed collection

• Pure - no side effects expected

;; Transform numbers to their squares
(map #(* % %) [1 2 3 4])
;; => (1 4 9 16)

;; Extract user IDs
(map :user-id users)
;; => ("alice" "bob" "charlie")

;; Transform with a function
(map str/upper-case ["hello" "world"])
;; => ("HELLO" "WORLD")

When NOT to use map:

The key insight is that map is lazy, so side effects inside map can lead to surprising results.

If you're doing side effects, you probably want doseq or run! instead.

;; This creates a lazy sequence but doesn't execute
(def squares (map #(do (println "Computing" %)
                       (* % %))
                  [1 2 3]))
;; => No output yet!

;; Realizing the sequence triggers computation
(doall squares)
;; Computing 1
;; Computing 2
;; Computing 3
;; => (1 4 9)

2. for - List Comprehensions

Purpose: Build a collection with complex iteration logic (filters, multiple bindings, let bindings).

Returns: A lazy sequence.

When to use:

• You need complex iteration with multiple sequences

• You want to filter while iterating

• You need let bindings inside the iteration

• You're translating a mathematical set notation

Characteristics:

• Lazy - only computes when realized

• Returns a sequence - the generated collection

• Declarative - reads like "for each X in Y, produce Z"

;; Cartesian product with filter
(for [x [1 2 3]
      y [4 5 6]
      :when (even? (+ x y))]
  [x y])
;; => ([1 5] [2 4] [2 6] [3 5])

;; Multiple bindings with let
(for [user users
      :let [email (:email user)
            domain (second (str/split email #"@"))]
      :when (= domain "example.com")]
  (:name user))
;; => ("Alice" "Bob")

;; Nested iteration
(for [feed feeds
      item (:items feed)]
  (process-item feed item))

;; Perfect for generating a deck of cards
(def suits #{:hearts :diamonds :clubs :spades})
(def ranks ["2" "3" "4" "5" "6" "7" "8" "9" "10" "J" "Q" "K" "A"])

(defn make-deck
  "Create a 52-card deck"
  []
  (shuffle
   (for [suit suits
         rank ranks]
     {:suit suit :rank rank})))

Rule of thumb: Use for when you need :when, :let, or multiple sequences. Use map for simple one-to-one transformations.

Generally, follow the same rules as map Don’t use this when you need to generate side effects.

;; Unnecessarily verbose
(for [user users]
  (:email user))

;; Better - Use map
(map :email users)

;; weird with side effects
=> (doall (map println [1 2 3]))
1
2
3
(nil nil nil)

3. doseq - Imperative Iteration for Side Effects

Purpose: Execute side effects for each element in a collection.

Returns: nil (you're not using the return value).

When to use:

• You're doing I/O (printing, file writes, network calls)

• You're mutating state (atoms, databases)

• You need multiple operations per item

• You need destructuring or complex bindings

Characteristics:

• Eager - executes immediately

• Returns nil - signals "I'm doing side effects"

• Imperative - like a for loop in other languages

;; doseq - Best for performing side effects for each result

;; Destructuring with complex logic with :keys
(doseq [{:keys [name email role]} users]
  (when (= role "admin")
    (send-admin-notification name email)))

;; Multiple operations per item
(doseq [order orders]
  (send-confirmation-email order)
  (update-inventory order)
  (log-transaction order))

;; Nested iteration
(doseq [feed feeds]
  (println "Processing" (:feed-id feed))
  (doseq [item (:items feed)]
    (save-to-db item)))

4. run! - Concise Side Effects

Purpose: Apply a single side-effecting function to each element.

Returns: nil.

When to use:

• You're calling one side-effect producing function per item

• You don't need destructuring

• You want concise code

Characteristics:

• Eager - executes immediately

• Returns nil - signals side effects

• Concise - one-liner for simple cases

run! is more concise for when you're just calling a single function

;; doseq - verbose
(doseq [alert alerts]
  (println alert))

;; Better with run! - concise
(run! println alerts)

;; Send email to each user
(run! send-email users)

;; Save each document
(run! save-to-db documents)

;; Close each connection
(run! #(.close %) connections)

When You Shouldn’t Use run!

;; Multiple operations - better with doseq
(doseq [user users]
  (send-email user)
  (log-activity user)
  (update-db user))

;; Destructuring
;; With run! you need a wrapper
(run! (fn [{:keys [name email]}]
        (send-email name email))
      users)

;; Better with doseq!
(doseq [{:keys [name email]} users]
  (send-email name email))

;; Conditional logic
;; With run! you need the logic in the function
(run! (fn [user]
        (when (active? user)
          (send-notification user)))
      users)

;; doseq is clearer
(doseq [user users]
  (when (active? user)
    (send-notification user)))

Summary

Understanding the different iteration constructs in Clojure is crucial for writing efficient and effective code.

Use map and for when transforming data with lazy evaluation and pure functions. For producing side effects, doseq and run! are better suited, providing eager execution and clear intent for side-effecting operations.

I hope this helps on your Clojure journey. It has definitely helped in mine.

Further Reading Resources:

• Clojure’s for

• Clojure map documentation

• Clojure doseq documentation

• Clojure doall documentation

• Iteration for Side Effects - Clojure Guides

• Clojure run! documentation

Validating URLs is one of those cases that is perfect for property-based testing. I happened to use fast-check and they have a pretty comprehensive page on why you would need property based testing in the first place. Please read it.

The idea is that you just don’t want to just test on a few hardcoded values, get some basic tests passing and call it a day. You want to test your application logic against a wide range of possibly valid data and ensure the application behaves correctly when provided with a wide range of data, especially examples that you wouldn’t think to check. Property based tests are a pretty elegant approach to this problem, since libraries like fast-check can take care of generating valid values, in this case URLs, and then feeding them to the tests and checking that the code behaves as expected.

Property based tests using fast-check looks like this:

import fc from 'fast-check';
import { validateUrl } from '../server/services/FetcherUtils.res.mjs';

describe("URL validation rejects invalid URLs", () => {
  it("should accept valid http and https URLs with domain names", async () => {
    const validUrl = fc.webUrl({
      validSchemes: ['http', 'https'],
      withFragments: true,
      withQueryParameters: true,
    });

    fc.assert(
      fc.property(validUrl, (url) => {
        const result = validateUrl(url);
        return result.valid === true && result.error === undefined;
      }),
      { numRuns: 20 }
    );
  });

  it("should reject URLs with invalid protocols", async () => {
    const invalidProtocolUrl = fc.webUrl({
        validSchemes: ['ftp', 'file', 'javascript', 'data', 'mailto', 'ws', 'wss'],
        withFragments: true,
        withQueryParameters: true,
      });

    fc.assert(
      fc.property(invalidProtocolUrl, (url) => {
        const result = validateUrl(url);
        return (
          result.valid === false &&
          result.error === "URL must use http or https protocol"
        );
      }),
      { numRuns: 20 }
    );
  });
});

The idea should be clear - Generate different kinds of URLs, both valid and invalid, and ensure your code handles it appropriately.

The functions used to generate different kinds of data should be mostly self-explanatory. In this example we are using fc.webUrl to generate different kinds of URLs specifying the schemes we want to allow

As an aside, the function being tested is imported from a compiled Rescript file. This is one of the great advantages of Rescript. It gives you all the benefits of a type-safe language, but it compiles to plain JS, so there is zero lock-in. You can even delete it and keep the compiled JS files. Anyway for this case, the nice part about this example is that you can use all the amazing testing tools and libraries already available in JS-land to test functions written in Rescript 🤩

Detour: What is a valid URL?

I came to this structure after finding out that my original code was not restrictive enough. The property tests were generating quite strange URLs that kept being accepted as valid. This right here is one of the strengths of using property-based tests. So what was going on?

This is the JavaScript compiled version of the code I started out with to accept and fetch URLs, basically deferring to the already existing URL interface in JavaScript.

Looks like an innocent and simple function but don’t be fooled. There are a lot of dangers lurking underneath. Read on for all the issues I found from this simple function

function fetchUrl(userInput) {
  const url = new URL(userInput);
  return fetch(url);
}

For full transparency, I asked an AI agent to document why some of the weird URLs that were being generated from the property tests were being accepted. I thought it best to write it down somewhere easy to reference and so that I don’t need to re-discover this again in the future.

JavaScript URL Constructor Quirks

The JavaScript URL constructor (based on the WHATWG URL Standard) accepts many URL formats that may be surprising or unexpected. This document catalogs these behaviors to help developers understand URL validation edge cases.

Key Takeaway: The URL constructor is very permissive and accepts many strings that don't look like traditional URLs. Always validate the parsed components (protocol, hostname, etc.) after construction.

Unexpected Formats Accepted

1. Single-Letter Schemes

The URL constructor accepts any single letter followed by a colon as a valid URL scheme.

new URL('A: ')
// ✅ Valid
// protocol: "a:"
// hostname: ""
// href: "a:"

new URL('Z:')
// ✅ Valid
// protocol: "z:"
// hostname: ""
// href: "z:"

new URL('x:anything')
// ✅ Valid
// protocol: "x:"
// hostname: ""
// pathname: "anything"

Why: Per the URL spec, any string matching the pattern [a-zA-Z][a-zA-Z0-9+.-]*: is a valid scheme.

Implication: Must explicitly check for allowed protocols (http/https) after parsing.

2. Schemes Without Authority

URLs don't require the // authority component.

new URL('mailto:user@example.com')
// ✅ Valid
// protocol: "mailto:"
// pathname: "user@example.com"

new URL('data:text/plain,Hello')
// ✅ Valid
// protocol: "data:"
// pathname: "text/plain,Hello"

new URL('javascript:alert(1)')
// ✅ Valid
// protocol: "javascript:"
// pathname: "alert(1)"

Why: Many URL schemes (mailto, data, javascript, etc.) don't use the // authority syntax.

Implication: Dangerous for security - must validate protocol to prevent XSS and other attacks.

3. Single Slash After Protocol

URLs with a single slash after the protocol are valid.

new URL('http:/example.com')
// ✅ Valid
// protocol: "http:"
// hostname: "example.com"
// pathname: ""

new URL('https:/path/to/resource')
// ✅ Valid
// protocol: "https:"
// hostname: "path"
// pathname: "/to/resource"

Why: The URL spec allows this - the authority component is optional.

Implication: May parse differently than expected - hostname might not be what you think.

4. Empty Authority

URLs can have an empty authority (no hostname).

new URL('http://')
// ❌ Throws TypeError: Invalid URL

new URL('http:///')
// ✅ Valid
// protocol: "http:"
// hostname: ""
// pathname: "/"

new URL('file:///')
// ✅ Valid
// protocol: "file:"
// hostname: ""
// pathname: "/"

Why: The spec allows empty hostnames for certain schemes.

Implication: Must check that hostname is not empty for http/https URLs.

5. Whitespace Handling

Leading and trailing whitespace is trimmed, but internal whitespace causes errors.

new URL('  http://example.com  ')
// ✅ Valid (whitespace trimmed)
// href: "http://example.com/"

new URL('http://example .com')
// ❌ Throws TypeError: Invalid URL

new URL('http://example.com/path with spaces')
// ❌ Throws TypeError: Invalid URL

Why: The spec requires trimming ASCII whitespace from the input.

Implication: Whitespace in the middle is invalid, but leading/trailing is silently removed.

6. Case Insensitivity

Schemes and hostnames are case-insensitive and normalized to lowercase.

new URL('HTTP://EXAMPLE.COM')
// ✅ Valid
// protocol: "http:"
// hostname: "example.com"
// href: "http://example.com/"

new URL('HtTp://ExAmPlE.cOm')
// ✅ Valid
// protocol: "http:"
// hostname: "example.com"

Why: DNS and URL schemes are case-insensitive per spec.

Implication: Always compare protocols and hostnames in lowercase.

7. Numeric Hostnames

Hostnames can be numeric (interpreted as IP addresses).

new URL('http://127.0.0.1')
// ✅ Valid
// hostname: "127.0.0.1"

new URL('http://2130706433')
// ✅ Valid (decimal IP representation)
// hostname: "2130706433"

new URL('http://0x7f000001')
// ✅ Valid (hexadecimal IP)
// hostname: "0x7f000001"

Why: The spec allows various IP address formats.

Implication: Must validate that hostname is not an IP address if you want to require domain names.

8. IPv6 Addresses

IPv6 addresses must be enclosed in brackets.

new URL('http://[::1]')
// ✅ Valid
// hostname: "[::1]"

new URL('http://[2001:db8::1]')
// ✅ Valid
// hostname: "[2001:db8::1]"

new URL('http://::1')
// ❌ Throws TypeError: Invalid URL

new URL('http://2001:db8::1')
// ❌ Throws TypeError: Invalid URL

Why: Brackets are required to disambiguate colons in IPv6 from port numbers.

Implication: IPv6 addresses without brackets are invalid.

9. Port Numbers

Port numbers can be specified but are optional.

new URL('http://example.com:8080')
// ✅ Valid
// port: "8080"

new URL('http://example.com:')
// ✅ Valid (empty port)
// port: ""

new URL('http://example.com:abc')
// ❌ Throws TypeError: Invalid URL

Why: Ports must be numeric or empty.

Implication: Empty port is valid but non-numeric ports are not.

10. Userinfo in URLs

URLs can contain username and password (deprecated for security).

new URL('http://user:pass@example.com')
// ✅ Valid
// username: "user"
// password: "pass"
// hostname: "example.com"

new URL('http://user@example.com')
// ✅ Valid
// username: "user"
// hostname: "example.com"

Why: The spec supports userinfo for backward compatibility.

Implication: Security risk - passwords in URLs are visible in logs, history, etc.

11. Fragment and Query Handling

Fragments (#) and queries (?) are always valid.

new URL('http://example.com#fragment')
// ✅ Valid
// hash: "#fragment"

new URL('http://example.com?query=value')
// ✅ Valid
// search: "?query=value"

new URL('http://example.com?#')
// ✅ Valid
// search: "?"
// hash: "#"

Why: Fragments and queries are standard URL components.

Implication: Always present, even if empty. Ensure to parse the whole URL not just the domain.

12. Relative URLs Require Base

Relative URLs need a base URL to resolve against.

new URL('/path/to/resource')
// ❌ Throws TypeError: Invalid URL

new URL('/path/to/resource', 'http://example.com')
// ✅ Valid
// href: "http://example.com/path/to/resource"

new URL('//example.com/path')
// ❌ Throws TypeError: Invalid URL

new URL('//example.com/path', 'http://base.com')
// ✅ Valid (protocol-relative URL)
// href: "http://example.com/path"

Why: Relative URLs need context to be resolved.

Implication: Must provide base URL for relative URLs. (This was absolutely new for me)

Security Implications

XSS (Cross-Site Scripting)

JavaScript URLs can execute code:

new URL('javascript:alert(1)')
// ✅ Valid but DANGEROUS
// protocol: "javascript:"

// ❌ DANGEROUS - Could execute JavaScript
element.href = userInput;

// ✅ SAFE - Validates protocol
const url = new URL(userInput);
if (url.protocol !== 'http:' && url.protocol !== 'https:') {
  throw new Error('Invalid protocol');
}
element.href = url.href;

Resolutions

I ended up discovering a better way to parse and validate a URL before blindly fetching it, especially if it comes from external sources.

// ❌ DANGEROUS - Accepts many unexpected formats
function fetchUrl(userInput) {
  const url = new URL(userInput);
  return fetch(url);
}

// Basic SSRF protection
// Avoid redirect to unintended locations e.g. internal resources
function isValidDomainName(hostname) {
  // Check it's not an IP address
  if (/^\d+\.\d+\.\d+\.\d+$/.test(hostname)) {
    return false;
  }

  // Check it's not IPv6 (in brackets)
  if (hostname.startsWith('[') && hostname.endsWith(']')) {
    return false;
  }

  // Check it contains at least one dot (has TLD)
  if (!hostname.includes('.')) {
    return false;
  }

  // Check it's not localhost
  if (hostname === 'localhost') {
    return false;
  }

  return true;
}

// ✅ SAFE - Validates protocol and hostname
function fetchUrl(userInput) {
  const url = new URL(userInput);

  // ✅ Only allow known-safe protocols
  if (url.protocol !== 'http:' && url.protocol !== 'https:') {
    throw new Error('Invalid protocol');
  }

  // Check hostname is not empty
  if (!url.hostname) {
    throw new Error('Hostname required');
  }

  // Validate not IP / Loopback / IPV6
  if (isValidDomainName(url.hostname)) {
    throw new Error('IP addresses not allowed');
  }

  return fetch(url);
}

Conclusion

As a developer working on web applications, knowing all the kind of URLs that exist and are considered valid should have been common knowledge. Unfortunately a lot of the strange valid URLs were new to me and I was learning most of the security vulnerabilities hiding behind a simple URL for the first time. For example, I had no idea about Server-side request forgery (SSRF) but fortunately now I am much wiser.

Adopting property-based testing helped me discover some huge security holes in my application and I hope this discovery and the examples can also help others avoid similar issues when developing their own web applications.

Rescript offers an excellent way to work with existing JavaScript code and integrated especially well with React. Each Rescript file compiles into a single JS file, making it easy to use in any JS project or to gradually convert small bits to Rescript. This made it ideal for this experiment.

This leads me to some of the lessons I have had to learn the hard way. I don’t see a lot of content about Rescript out there and I had to discover some of this myself, so just writing it down for anyone else who might run into similar issues.

They are all about interoperating with JS so that’s an area that you need to be careful with in general. Also a lot of this code is AI-generated, so it might reflect the quality or age of the code it was trained with. Also Rescript is changing quite a lot with the recent v12 release being quite aggressive with deprecating old stuff, so also watch out when using AI agents with Rescript.

1. Always verify the AI output 🪄

I saw the AI agent sprinkling a few Obj.magic calls arond and was left wondering what this function is and what it does. There isn’t much information on this on the official docs, so it seems like an API that is not encouraged or deprecated.

Anyway I asked the Agent to explain:

Obj.magic is a ReScript function that performs an unsafe type cast - it tells the type system "trust me, I know what I'm doing" and bypasses type checking.

Here’s an example

let x: int = 42 

// Obj.magic can convert ANY type to ANY other type 
// Compiles, but DANGEROUS!
let y: string = Obj.magic(x)

Yikes! Looks dangerous, and looks exactly like the kind of thing that beats the whole point of using a strongly typed language in the first place.

This was the actual code in my case.

// types are not known in advance
let validateBooleanQueryFromJS = (
  ~mustTerms: 'a,
  ~shouldTerms: 'b,
  ~mustNotTerms: 'c,
) => {
   // Check if inputs are arrays
  if !Array.isArray(mustTerms) {
    {"valid": false, "error": Some("mustTerms must be an array")}
  } else if !Array.isArray(shouldTerms) {
    {"valid": false, "error": Some("shouldTerms must be an array")}
  } else if !Array.isArray(mustNotTerms) {
    {"valid": false, "error": Some("mustNotTerms must be an array")}
  } else {
    // Cast to array after the checks
    let mustTermsArray: array<string> = Obj.magic(mustTerms)
    let shouldTermsArray: array<string> = Obj.magic(shouldTerms)
    let mustNotTermsArray: array<string> = Obj.magic(mustNotTerms)

    // ...
    // Further processing of the arrays
}

And this is how the function is called from the JS side

 const result = validateBooleanQueryFromJS(mustTerms || [], shouldTerms || [], mustNotTerms || []);

So I finally understood why the Obj.magic call was included. Since there are no explicit types being specified for mustTerms, shouldTerms and mustNotTerms in the validateBooleanQueryFromJS function, it is possible to call the function passing any data type.

We are validating the data through the explicit Array.isArray calls, then casting each of the arrays to an array<string> type afterwards when we have validated that indeed the data passed in is an array.

There are a couple of ways we can deal with this.

The first option is the more straightforward one: Do the same validation, but on the JS side, then using the concrete array<string> type for the function parameters.

This basically means moving the array checks to before calling the Rescript method validateBooleanQueryFromJS from the JavaScript side

The parameter types can now be written as follows:

let validateBooleanQuery = (
  ~mustTerms: array<string>,
  ~shouldTerms: array<string>,
  ~mustNotTerms: array<string>,
) => {
    // No need to check if inputs are arrays
    // ...
    // Further processing of the arrays

And called from the JS side after doing the validations there

// Option 1: Simple and straightforward validation
// Less clarity on which variable has the wrong type
const terms = [mustTerms, shouldTerms, mustNotTerms];
if (!terms.every(Array.isArray)) {
  return res.status(400).json({ error: 'Terms must be arrays' });
}

const result = validateBooleanQuery(mustTerms, shouldTerms, mustNotTerms);

// Option 2: More verbose and more explicit
if (!Array.isArray(mustTerms)) {
  return res.status(400).json({ error: 'mustTerms must be an array' });
}
if (!Array.isArray(shouldTerms)) {
  return res.status(400).json({ error: 'shouldTerms must be an array' });
}
if (!Array.isArray(mustNotTerms)) {
  return res.status(400).json({ error: 'mustNotTerms must be an array' });
}

This could be convenient, but if there is another place where the same function needs to be called and the same validations need to be done again, it can get hectic.

The second option is to keep the unsafe type conversions with %identity which is the supported syntax. To be clear, this is essentially the same as Obj.magic and is generally not recommended to unsafely convert between types. Reserve this for when there’s no other option.

 // Use %identity for unsafe conversion from one type to another type.
external convertToArray: 'a => array<string> = "%identity"

let validateBooleanQueryFromJS = (
  ~mustTerms: 'a,
  ~shouldTerms: 'b,
  ~mustNotTerms: 'c,
) => {
   // Check if inputs are arrays
  if !Array.isArray(mustTerms) {
    {"valid": false, "error": Some("mustTerms must be an array")}
  } else if !Array.isArray(shouldTerms) {
    {"valid": false, "error": Some("shouldTerms must be an array")}
  } else if !Array.isArray(mustNotTerms) {
    {"valid": false, "error": Some("mustNotTerms must be an array")}
  } else {
    // Use convertToArray to safely cast after verifying types
    let mustTermsArray = convertToArray(mustTerms)
    let shouldTermsArray = convertToArray(shouldTerms)
    let mustNotTermsArray = convertToArray(mustNotTerms)

    // ...
    // Further processing of the arrays
}

So you can choose what works best for you. Both are valid options and it helps to know the tradeoffs you are making with each method.

So I’m watching the content from the AI agent more closely, and trying to learn from the agent rather than just accepting whatever code it provides.

Check out this excellent post that is required reading for anyone working with AI-generated code: Treat AI-Generated code as a draft

2. Conversion between Rescript types and JavaScript

Some parts of Rescript are very nice such as the Result type, which encodes an operation that could succeed (Ok) or fail (Error). It’s particularly helpful for avoiding throwing exceptions and handling errors more gracefully with pattern matching.

type result<'ok, 'error> = Ok('ok) | Error('error)

let divide = (a, b) => {
   if b == 0 {
     Error("Division by zero")
   } else {
     Ok(a / b)
   }
 }

switch divide(10, 0) {
| Error(err) => Console.log("Nice try! Error! " ++ err)
| Ok(result) => Console.log("Success: " ++ Int.toString(result))
}

However, trying to return the Result type from a function and using this directly from JS is a bit weird. For example this divide function compiles to this JavaScript code

function divide(a, b) {
  if (b === 0) {
    return {
      TAG: "Error",
      _0: "Division by zero"
    };
  } else {
    return {
      TAG: "Ok",
      _0: Primitive_int.div(a, b)
    };
  }
}

And using this does seem a bit strange and inconvenient. Reaching into a field called result._0 definitely seems like an API that was built not to be used directly

const { divide } = require('path/to/compiled_js');

const result = divide(10, 2);

if (result.TAG === "Ok") {
  console.log("Success:", result._0);
} else if (result.TAG === "Error") {
  console.log("Error:", result._0);
}

So in such cases, we could create and export a new function for use from JavaScript. This function can return a custom object, providing a more user-friendly experience when using the function from JS

// Export a wrapper for JavaScript - return a plain object instead of result
let divideFromJS = (a, b) => {
  switch divide(a, b) {
  | Ok(value) => {
      "success": true,
      "value": value,
    }
  | Error(msg) => {
      "success": false,
      "error": msg,
    }
  }
}

And this is much nicer to use from JS

const { divideFromJS } = require('path/to/compiled_js');

const result = divideFromJS(10, 2);

if (result.success) {
  console.log('Result:', result.value);
} else {
  console.log('Error:', result.error);
}

So what have I learned from all this?

1. Rescript provides a nice experience for writing type safe JavaScript code with no runtime errors. However the interoperability with JavaScript sometimes can sometimes be hectic, and you can customize your functions and return types for a better experience.

2. Always review AI-generated code. Check for unsafe or deprecated APIs usage and ensure whatever it generates fits your needs

3. Languages like Rescript help to tame JavaScript. It’s hard to go back to writing plain JS after experiencing a compiler that gives you confidence when shipping

Back to playing with some typed languages 🤖

Up until very recently I believed this was all hype. No way this could happen.

There have always been proclamations of new technologies like AI taking jobs. Most of the time dismissed as fantasy, most takes filled with a hint of denial, not willing to accept reality.

Maybe the frontend development jobs, or the junior developers jobs, or some other job that’s not mine.

The time of reckoning is here.

All of the models are available for pretty much free or a very low cost and the future does not look good for software engineering careers.

AI can now do the job of several developers. With tools evolving at breakneck speed, we can now extract maximum performance from the best AI tools, providing just the needed context and making them perform even the most complicated tasks. It’s just a matter of time before the models start wrecking carnage on the job market.

So what can we do?

Get better at the craft. AI makes mistakes. It is more valuable to be able to spot the LLM’s mistakes especially when AI is generating a huge volume of code. The only way to get better is from gaining real experience and diving deeper to be more familiar with the tools, languages and frameworks you use. AI can still help with a lot of the development but it helps to know where to check for the occasional errors.

Gain complementary skills. Building working long-lived software is not just about conjuring pretty algorithms. Keeping a system running for years while handling evolving requirements from paying customers requires more than just ninja coding skills. Operating at scale with larger data sets requires skills like extracting insights from data, optimization to enable efficient use of resources, investigation and debugging when the occasional incident or unexpected errors occur, working in a team, and so on.

Get closer to the business and product. Understand the real needs of customers. Get better at converting these into viable product specs and implementing them within the constraints that exist in your particular business. This is harder to outsource to AI than just implementing code.

Ultimately this is a double edged sword. What are some of the positives?

Massive Productivity Unlock I have never been more productive especially with personal projects.

With AI tools, I am learning faster and developing features from start to finish faster than ever. A single person turbocharged with AI tools is now more capable than ever, and it would be a huge self-inflicted mistake not to take advantage of this huge productivity boost.

There are no more blockers, even on personal projects, where there are real constraints like not having experienced colleagues to bounce ideas off.

No limit to learning.
I have always been fascinated by some of the niche languages with good ideas, but which still remain unpopular, like Ocaml. Previously it was always a disadvantage working with such esoteric languages. There aren't as many resources to learn from, not many example projects, not the biggest communities, and lots of other issues that don’t face the more popular languages. Right now with the AI tools available today none of those matter. You can learn whatever you want with no limitations.

What next?

This is just my impression of things as I see them. I have no idea how it ends up but the best strategy seems to be trying to keep up, always be learning and hope to survive whatever comes next ✌️

This got me wondering, can we replicate this result ourselves? 🤔

Let’s find out.

Language of Choice

We will go with F# to do this.

I’ve been checking it out recently and found it to be a really nice concise language (like most other functional languages), with a really helpful type system and an amazing development experience. I love the ability to interactively evaluate expressions right inside the editor. Reminds me a lot of Clojure.

This article: Why is F# code so robust and reliable? does a great job of explaining its amazing features better than I could, so please go and check it out.

I like it so far and want to use it more, so I’m taking every chance I get to build more projects with it 👷🏾

Understanding the history file

The first thing we need to do is to get where these historical commands are written and try to read the file ourselves.

In my case (using zsh and oh-my-zsh) I found that the history file is stored in the $HISTFILE environment variable. This is nice since we can just provide the same environment variable to our program.

Let’s see how it looks like with cat $HISTFILE

We can see in the small sample here that there are some blank lines mixed in with others that contain a command, along with some other content. Let’s figure it out.

It looks like the lines with the commands start with a colon : and a space, then a number that looks like a unix timestamp, and another colon then 0
Then we have a semicolon ; and finally we have the actual command.

According to ChatGPT this the meaning of each part of the line

1. : <epoch_timestamp>:

   • This is a Unix epoch timestamp (the number of seconds since January 1, 1970). It represents when the command was executed.

2. : <elapsed_time_in_seconds>:

   • This is the amount of time (in seconds) that the command took to execute. It's the difference between the time the command started and when it finished.

3. ;command:

   • After the semicolon (;), the actual command that was executed in the shell appears. This is the command as the user typed it.

Now that we understand what we have to deal with, let’s setup the project.

Project Setup

If you want to follow along ensure you have .NET installed. As of the time of writing, the most current version is .NET8.

Once everything is setup you should have the dotnet command

❯ dotnet --version
8.0.104

Let’s use the dotnet CLI to create a new console app and call it HistoryStats. We also need to specify that we want to use F# as the language.

dotnet new console -lang "F#" -o HistoryStats

It generates a console app with this directory structure

❯ tree
.
├── HistoryStats.fsproj
└── Program.fs

1 directory, 2 files

We will add all the subsequent code to the Program.fs file.

To make sure everything is working, run the app with dotnet run and you should see the following message printed from the default program

"Hello from F#"

Now that we have the environment ready, let’s get started.

The eventual goal is to run dotnet run $HISTFILE and it should give us a ranked list of the most commonly used commands in our terminal.

Reading the History File

This is the first iteration of a program to read and print out each line from the history file, which is provided as a command line argument

open System
open System.IO

let commandsByFrequency historyFile =
    File.ReadLines(historyFile)
    |> Seq.iter (fun line -> printfn "%s" line)

[<EntryPoint>]
let main argv =
    let args = Environment.GetCommandLineArgs()

    // Optional: Print all the arguments
    printfn "Command-line arguments: %A" args

    match args with
    | [| _app; historyFile; |] ->
        printfn "First argument: %s" historyFile
        readFileLines historyFile

    | _ ->
        printfn "Usage: dotnet run <historyFile>"

    0

The main part to focus on is the commandsByFrequency function, which reads the provided file line by line. We then use Seq.iter to iterate and print out each line in the file. We will add most of the functionality to this function.

In the main function, which will be the entrypoint, we also have some pattern matching to ensure we only try to process the file if the program has been run with the first argument, which should be the history file.

If the file path is not provided, we print a message showing that we require an argument to be provided and how to provide it.

Now we can run our application, providing $HISTFILE as the argument, and it should print out each line of the history file.

dotnet run $HISTFILE

Parsing the History File

On my system, printing out all the lines in the history file is quite noisy, so let’s start processing those lines to get to the interesting part.

We will focus on the commandsByFrequency function to do all the processing we need.

First, we need to skip the blank lines before continuing with the processing

let commandsByFrequency filePath =
    // Returns an enumerable over the lines in the file
    File.ReadLines(filePath)
    |> Seq.choose (fun line -> if String.IsNullOrWhiteSpace line then None else Some line)
    |> Seq.iter (fun line -> printfn "%s" line)

If you run the app now you should see that we no longer print out the blank lines in the file.

We will make heavy use of the Seq module, which is roughly similar to the Enumerable module in Ruby. It has a ton of useful functions for processing collections of data. It is lazy by default and only processes individual sequence elements as required, making it a nice tool for our needs.

Seq.choose deserves a special mention since it makes it so easy to separate the lines we want to keep and the ones to discard. It takes a function that should return None if we want to skip the item or Some x if we want to keep the item x

I found it to be quite elegant and we will use it again in the next section.

Extracting the Command from a History Line

Now that we have established a pattern for iterating over the lines in the file, let’s start parsing them to extract the info we need.

Just a reminder that this is how a line in the history file looks:
: 1726414019:0;brew info mongod

To extract the command from a non-blank line, let’s add a new function that should do 2 things:

1. Get the command, which starts immediately after the first semicolon i.e. ;

2. Extract the first part of the command, without the arguments

let parseHistoryLine (line: string) =
    if line.StartsWith(":") then
        let semiColonIndex = line.IndexOf(';')
        // 1. Get the command i.e. everything after the semicolon
        let fullCommand = line.Substring(semiColonIndex + 1)
        // 2. Split by space and get the first part of the command
        let command = fullCommand.Split([| ' ' |]) |> Array.head
        Some command
    else
        None

We will use the same pattern of using a function that returns an option, to determine which lines to keep or discard, then passing this function to Seq.choose

We return Some command to indicate that want to keep the line and None to indicate that we want to discard the line.

By now we already know how to use this command in our processing pipeline:

let commandsByFrequency historyFile =
    File.ReadLines(historyFile)
    // Take only the non-blank lines
    |> Seq.choose (fun line -> if String.IsNullOrWhiteSpace line then None else Some line)
    // Extract the command from the line
    |> Seq.choose (fun line -> (parseHistoryLine line))
    // Optional: Print out the commands we will process
    |> Seq.iter (fun command -> printfn "%s" command)

Now that we have this pattern established, let’s take advantage of it to now finally get the most frequently used commands.

More Seq Magic

All that’s let for us to do is to find out how many times each command occurs. Once we have that, we can sort the sequence of commands in descending order, then get the top commands based on the frequency of occurrence.

Let’s write out the whole function then go through the important parts

let commandsByFrequency count historyFile =
    // Returns an enumerable over the lines in the file
    File.ReadLines(historyFile)
    // Take only the non-blank lines
    |> Seq.choose (fun line -> if String.IsNullOrWhiteSpace line then None else Some line)
    // Extract the command from the line
    |> Seq.choose (fun line -> (parseHistoryLine line))
    // Group by command -> (command, seq of commands)
    |> Seq.groupBy id
    // Count occurrences
    |> Seq.map (fun (command, occurrences) -> (command, Seq.length occurrences))
    // Sort by the count in descending order
    |> Seq.sortByDescending snd
    // Take the top `count` elements
    |> Seq.take count

The first change is the addition of a new count parameter, which represents the number of commands we want to return.

The first addition is Seq.groupBy

Applies a key-generating function to each element of a sequence and yields a sequence of unique keys. Each unique key contains a sequence of all elements that match to this key.

The key-generating function we provide is the id function, which simply returns whatever is provided to it as an input.

 id 12345     //  Evaluates to 12345
 id "command"  //  Evaluates to "command"

In our case, what it does is to group the commands by the command itself. It returns a tuple containing 2 items, where the first item is the command itself, and the second is each occurrence of the command in the provided sequence of commands.

This is the sample output we would get if we printed out the result at this point in the program

("omz_history", seq ["omz_history"])
("gds", seq ["gds"])
("ggpush", seq ["ggpush"])
("zsh_stats", seq ["zsh_stats"; "zsh_stats"; "zsh_stats"])

Now that we have the command and a sequence of all the occurrences, then we can count the occurrences of each command. This is what happens in the next step:

// Count occurrences
Seq.map (fun (command, occurrences) -> (command, Seq.length occurrences))

This takes in the tuple returned from the groupBy and transforms it into a new tuple containing the command and the frequency i.e. number of times it occurs.

So it would transform the previous input into something like this:

("omz_history", 1)
("gds", 1)
("ggpush", 1)
("zsh_stats", 3)

And now that we have the counts of each command, we can now sort the commands by the count in descending order, which is the second element of the tuple.

// Sort by the count in descending order
|> Seq.sortByDescending snd

snd is a function that returns the second element of a tuple, and we use it here to get the count and use it to sort the commands.

And now that we have the sorted commands, the final part is to return the top commands by the count.

// Take the top `count` elements
|> Seq.take count

And just like that, we have our very own custom zsh_stats clone

Wrapping up

Now we can take a look at the whole program

open System
open System.IO

let parseHistoryLine (line: string) =
    if line.StartsWith(":") then
        let semiColonIndex = line.IndexOf(';')
        // 1. Get the command i.e. everything after the semicolon
        let fullCommand = line.Substring(semiColonIndex + 1)
        // 2. Split by space to get the first part of the command
        let command = fullCommand.Split([| ' ' |]) |> Array.head
        Some command
    else
        None

let commandsByFrequency count historyFile =
    // Returns an enumerable over the lines in the file
    File.ReadLines(historyFile)
    // Take only the non-blank lines
    |> Seq.choose (fun line -> if String.IsNullOrWhiteSpace line then None else Some line)
    // Extract the command from the line
    |> Seq.choose (fun line -> (parseHistoryLine line))
    // Group by command -> (command, seq of commands)
    |> Seq.groupBy id
    // Count occurrences
    |> Seq.map (fun (command, occurrences) -> (command, Seq.length occurrences))
    // Sort by the count in descending order
    |> Seq.sortByDescending snd
    // Take the top `count` elements
    |> Seq.take count


[<EntryPoint>]
let main argv =
    let args = Environment.GetCommandLineArgs()

    match args with
    | [| _app; historyFile |] ->
        printfn "History file: %s" historyFile
        let result = commandsByFrequency 10 historyFile

        for (cmd: string, count) in result do
            printfn "Command: %s \tCount: %d" cmd count

    | _ -> printfn "Usage: dotnet run <historyFile>"

    0

The only addition is that we now print out the results of the commandsByFrequency function. We will default to getting the top 10 results for now.

And now for the moment of truth, let’s compare our program’s output and and the zsh_stats command output

There’s an off by one error with the ls result which returns 166 in our command vs 165 in zsh_stats but they are mostly similar and it gets the job done.

via GIPHY

Conclusion

In summary, we have gone through replicating the zsh function zsh_stats by writing a custom program in F#.

It has proven to be capable enough to replicate the original function, and more importantly, has helped us learn more about operating on sequences of data in F#.

PS:

If you are interested in how the original zsh_stats function works, here you go

❯ which zsh_stats
zsh_stats () {
        fc -l 1 | awk '{ CMD[$2]++; count++; } END { for (a in CMD) print CMD[a] " " CMD[a]*100/count "% " a }' | grep -v "./" | sort -nr | head -n 20 | column -c3 -s " " -t | nl
}

You can access the full example in the GitHub repository

To access it you just need to run zsh_stats and it will produce a ranked list of the top 20 commands and how many times they have been run.

This is the output I get from my terminal

A lot of those are command aliases from oh-my-zsh plugins, most of them from the git plugin. The top command there is a shortcut for git commit -m which is just a lot to type out every time I need to create a commit message.

This cheatsheet contains some more convenient shortcuts and the inbuilt oh-my-zsh plugins contain even more awesomeness! I always discover something new there.

Just thought this is a really cool feature. oh-my-zsh is pretty amazing 🤩

I have been using this custom implementation:

defmodule MyModule do
 use GenServer

  def report(server) do
    GenServer.call(server, :report)
  end

  # More functions

  @impl true
  def handle_call(:report, _from, state) do
    # No modifications. Return the current state
    {:reply, state, state}
  end
end

It is all good and it works. However, it turns out this is all I needed to do

{:ok, pid} = MyModule.start_link([])
# ... some more operations...
:sys.get_state(pid)

Well, it does say right there in the get_state docs that the whole purpose is to help the users not to reimplement it.

Now I know and so do you 😀

More resources:

https://hexdocs.pm/elixir/GenServer.html#module-debugging-with-the-sys-module

https://www.erlang.org/doc/man/sys.html#module

This is my (failed) attempt to install Ruby 2.2.10 natively on MacOS Ventura 13.4. It is very unlikely that anyone else out there needs this specific combination, but if you do, please go ahead and enjoy the content.

I'll be using rbenv to install Ruby as it's my preferred tool for managing multiple Ruby versions at once.

Using Docker would probably be the better way to go but Docker on Mac is way slower than Docker on Linux, so be prepared to wait longer for everything.

Following this guide may or may not work for other Ruby versions, or other MacOS versions. Use at your own risk.

Install OpenSSL 1.0

This is the first requirement when trying to install this Ruby version as it was the supported OpenSSL version at the time of the Ruby release.

Trying to install Ruby 2.2 will initially lead to this error message, which is the first issue we will try to resolve

rbenv install 2.2.10

To resolve this, we'll start by first installing the main dependencies - OpenSSL.

The current OpenSSL version at the time of writing is 3.1.2.1, but Ruby versions lower than 2.3 require OpenSSL 1.0. Since OpenSSL 1.0 is now deprecated, it has since been removed from Homebrew.

The best way I found to install it is through this repo and the command to do so is:

brew install nutcracker/tap/openssl@1.0

However, I quickly ran into this issue when installing:

I finally found the solution in this GH issue from @rhanton, which was to run the `brew install` command with the debug flag and ignore the error when it pops up and asks for user input

brew install -d nutcracker/tap/openssl@1.0

Configure Ruby Install Dependencies

We now need the `rbenv install` command to find the OpenSSL 1.0 libraries when installing, so we'll need to configure a few things in the terminal

export RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(brew --prefix openssl@1.0)"
export optflags="-Wno-error=implicit-function-declaration"

Install Ruby 2.2

rbenv install 2.2.10

And we will run into one more error:

There is a wrong use of the `rm` command somewhere in the install command and we'll need some more manual intervention to fix it.

I found the solution in this discussion in the ruby-build repository, which involves removing the problematic command from the config.status file

With that, we should finally have Ruby 2.2.10 working 🎉 ... but the problems are not all gone yet

Can you live with no gems?

The main issue we now have to deal with is being unable to install gems. The first thing we need to install to be able to use any gem is bundler

The latest version of bundler cannot be installed on Ruby versions before 2.3, so we will need this workaround to install an older version

gem install bundler -v '~>1'

Even with this latest workaround, there is still a final unresolved issue when trying to install any gems, which takes us back to the OpenSSL issue. Now we can't even install bundler since we can't connect to rubygems over SSL 🫠

I still haven't been able to resolve this but if you have any pointers please leave a comment!

And this is how our adventure ends...in disappointment. After all this effort we have a barely usable old version of Ruby installed. If there ever was a perfect use case for Docker this is it. It was still fun giving it a try 😀

Please try to use supported versions of the software you use whenever possible

1. When a user visits the homepage, they should see a list of posts and should be able to click on any of the posts to be taken to the blog post's page.

2. When a user visits a blog post page, they should see the blog details i.e. title and content, and a list of recommended posts at the bottom of the page.

3. Clicking on a recommended blog post's link should take the user to that post's page.

Now that we've figured out the requirements, let's proceed to build the app.

Building the App

We will need 3 components for our app: the home page, the blog post component and the app's root component, which decides which page to render depending on the route. For this example, we'll fetch the posts from a local file rather than making network requests.

Let's start off with the root component, which is where we'll mount the app to a specific DOM node.

import React from "react"
import ReactDOM from "react-dom"
import { BrowserRouter as Router, Route, Switch } from "react-router-dom"
import Home from "./Home"
import PostDetails from "./PostDetails"
import "./index.css"

const App = () => (
  <Router>
    <Switch>
      <Route exact path="/" component={Home} />
      <Route path="/posts/:postId" component={PostDetails} />
    </Switch>
  </Router>
)

ReactDOM.render(<App />, document.getElementById("root"))

We're using the ​​react-router-dom package to render either the Home or PostDetails component depending on the route. A request with a URL such as /posts/2 will be routed to the PostDetails component and the postId variable will take the value of 2.

Let's look at the Home component, which will act as our homepage:

import React from "react"
import { Link } from "react-router-dom"
import { posts } from "./posts"
import "./Home.css"

const Home = () => {
  return (
    <div className="App">
      <header className="App-header">
        <h1 className="App-title">Blog Posts</h1>
      </header>
      <div>
        {posts.map(post => (
          <div key={post.id}>
            <Link to={`/posts/${post.id}`}>
              <h4>{post.title}</h4>
            </Link>
          </div>
        ))}
      </div>
    </div>
  )
}

export default Home

In this file, we fetch posts locally from posts.js and use the Home component render a list of links to each of the posts. Each post has an id attribute which is used to construct the URL using the Link helper from react-router-dom​

This is how our sample ​post data looks:

export const posts = [
  {
    id: 22,
    title: "Welcome to React",
    body: `React makes it painless to create interactive UIs.
      Design simple views for each state in your application,
      and React will efficiently update and render just the
      right components when your data changes.
      Declarative views make your code more predictable and easier to debug.`,
  },
  {
    id: 23,
    title: "An Introduction to React Router",
    body: `There are three types of components in React Router:
          router components, route matching components, and navigation components.
          All of the components that you use in a web application should be
          imported from react-router-dom.`,
  },
  {
    id: 24,
    title: "Welcome to Redux",
    body: `It helps you write applications that behave consistently,
          run in different environments (client, server, and native), and are easy to test.
          On top of that, it provides a great developer experience, such as live code editing
          combined with a time traveling debugger.`,
  },
]

export const recommendedPosts = [
  {
    id: 30,
    title: "Angular",
    body: `Learn one way to build applications with Angular and reuse your code
          and abilities to build apps for any deployment target.
          For web, mobile web, native mobile and native desktop.`,
  },
  {
    id: 31,
    title: "Ember.js",
    body: `Ember.js is built for productivity. Designed with developer ergonomics in mind,
          its friendly APIs help you get your job done—fast.`,
  },
  {
    id: 32,
    title: "Backbone.js",
    body: `Backbone.js gives structure to web applications by providing models with
          key-value binding and custom events, collections with a rich API of
          enumerable functions, views with declarative event handling, and connects
          it all to your existing API over a RESTful JSON interface.`,
  },
]

export const allPosts = posts.concat(recommendedPosts)

The final piece of the puzzle is the ​​PostDetails.jsx component:

3

Link to PostDetails.jsx

In this component, we'll store the post's details in the component state, and all the values are initially empty. To fetch the post's data, we'll need to extract the postId parameter from the URL. ​​​​​​​​​​​​​​react-router-dom stores the route params in the component's props and it enables us to access this parameter via this.props.match.params.postId. Once we have the postId, we then find the post with a matching ID in the posts array, and set it in the component state. This is handled in thefetchPostData function.

In the ​​render method, we show the post's details and a Recommeded Posts section. When you click on one of the posts in this section, you should be taken to that post's URL and shown that post's details.

You can try out the app in it's current state at CodeSandbox and also view the live code in the Live Editor

However, we have a bug lurking somewhere in our code. Try clicking on one of the recommended posts and see what happens.

Figuring out the Problem

If you haven't tried out the live app, the issue we have is that when we're in a blog post's page, clicking on a recommended post doesn't seem to do anything. If you're keen though, you'll notice that the URL does change but the page content doesn't change 🤔

When you click on a recommended blog post link, React Router re-renders the PostDetails route component with a new postIdparameter. This parameter is passed to the component as props. We can be sure the component does receive new props since the URL changes. But why does the component fail to display the updated data?

The root of the issue here is that we fetch data only on componentDidMount which is not called when a component re-renders.

What we need to do to solve the problem is to fetch new post data when we detect that the postId parameter has changed. This is where componentDidUpdate comes in.

Solution

Here's a comment I came across in the react-router issue tracker that let me down the right path.

You probably want to just respond to prop changes with componentDidUpdate to refetch data when the same route is active but with different parameters

Ryan Florence

Therefore, to fix the issue, we have to add componentDidUpdate, where we'll check if the postId has been updated and then fetch the details of the new post. Once we have the new details and store them in the component state, the component re-renders and shows the updated content.

Here's the small addition we need to make to fix the bug:

componentDidUpdate(prevProps) {
    const {
      match: {
        params: { postId }
      }
    } = this.props;
    const prevPostId = prevProps.match.params.postId;
    if (prevPostId !== postId) {
      this.fetchPostData(postId);
    }

The componentDidUpdate lifecycle method takes in the previous props as the first argument. We extract the postId from the previous props, and compare that with the postId from the current props. If there's a change, we fetch new data using the current postId parameter. This helps us update the page content when the component re-renders with a new postId parameter, and this fixes our bug!

You can try out the new bug-free version on CodeSandbox and check out the code on the Live Editor. If you're interested in the accompanying code samples, you can access them on GitHub.

And thus ends the story of how I learned how to use componentDidUpdate on a real practical problem. This was an interesting discovery for me and I hope you've learnt something new too!

Cover Image: “Those Who Affected Me” by Malin Bobeck, CC BY 2.0