Weird Arcade Tips and Tricks

Useful Things

Memorizing FeatureSets

How I Imagined It

2 Queries

flowchart LR

d1[(Data Source 1)]
d2[(Data Source 2)]

fs1[FeatureSet 1]
fs2[FeatureSet 2]

d1 -->q1{query}--> fs1
d2 -->q2{query}--> fs2

subgraph f[For Loop]
f1[Feature 1]
f2[Feature 2]
f3[...]
fn[Feature n]
f1 & f2 & f3 & fn --> filt{{filter}} 
end

fs1 -.- f
fs2 -..- filt

filt --> output

Let’s think about an example. Say I’ve got a layer of US Counties, and for each county, I want to get some associated statistic from their corresponding state.

The way it feels when you write a query like that is like this: query each data source, then do a looping filter. Right?

How it Actually Works

n + 1 Queries

flowchart LR

d1[(Data Source 1)]
d2[(Data Source 2)]

fs1[FeatureSet 1]

d1 -->q1{query}--> fs1

subgraph f[For Loop]
f1[Feature 1]
f2[Feature 2]
f3[...]
fn[Feature n]
f1 & f2 & f3 & fn --> filt{query} 
end

fs1 -.- f

d2 -..- filt

filt --> output

Turns out that’s not how Arcade executes, though.

It sees the second FeatureSet linking back to a datasource, but doesn’t query that source until it needs to. In this example, the second data source isn’t actually asked for until we’re filtering it inside our loop.

So, for 3,144 features in the Counties layer, we end up with 3,145 queries in total.

But there are only 51 features in the states layer! (DC)

That means that 3,093 of our queries are a duplicate. If the system could somehow know it already had the information to re-use, we could really cut back on this.

The Function

function Memorize(fs) {
  var temp_dict = {
    fields: Schema(fs)['fields'],
    geometryType: '',
    features: []
  }

  for (var f in fs) {
    var attrs = {}

    for (var attr in f) {
      attrs[attr] = Iif(TypeOf(f[attr]) == 'Date', Number(f[attr]), f[attr])
    }

    Push(
      temp_dict['features'],
      {attributes: attrs}
    )
  }

  return FeatureSet(Text(temp_dict))
}

Unfortunately, we can’t tell the system to do that. But what we can do is force the entire FeatureSet into our browser’s memory. Then it won’t need to query anything!

Normally, defining the output for a Data Expression can be tedious, you have to write out every single field. Here, though, we just use the function Schema to grab the existing fields.

Then, one feature at a time, we populate an array of features, which we convert into a FeatureSet. This effectively severs the connection between the data and its source, and we are then free to work with the data without any further communication with the server.

Comparison

Example Dashboard

// memorize function
function Memorize(fs) {
  var temp_dict = {
    fields: Schema(fs)['fields'],
    geometryType: '',
    features: []
  }

  for (var f in fs) {
    var attrs = {}

    for (var attr in f) {
      attrs[attr] = Iif(TypeOf(f[attr]) == 'Date', Number(f[attr]), f[attr])
    }

    Push(
      temp_dict['features'],
      {attributes: attrs}
    )
  }

  return FeatureSet(Text(temp_dict))
}

// output dictionary
var out_dict = {
  fields: [
    {name: 'step', type: 'esriFieldTypeInteger'},
    {name: 'duration', type: 'esriFieldTypeInteger'},
    {name: 'method', type: 'esriFieldTypeString'}
  ],
  geometryType: '',
  features: []
}

// start step count, timer
var i = 0
var start = Now()

Push(
  out_dict['features'],
  {
    attributes: {
      step: i,
      duration: DateDiff(Now(), start),
      method: 'Traditional'
    }
  }
)

// states and counties
var states = FeatureSetByPortalItem(
  Portal('https://arcgis.com'),
  '8c2d6d7df8fa4142b0a1211c8dd66903',
  0,
  ['STATE_FIPS', 'POPULATION'],
  false
)

var counties = FeatureSetByPortalItem(
  Portal('https://arcgis.com'),
  '3c164274a80748dda926a046525da610',
  0,
  ['NAME', 'STATE_FIPS', 'POPULATION'],
  false
)

// loop through counties
for (var c in counties) {
  // get parent state feature
  var the_state = First(Filter(states, `STATE_FIPS = '${c['STATE_FIPS']}'`))

  Push(
    out_dict['features'],
    {
      attributes: {
        step: i,
        duration: DateDiff(Now(), start),
        method: 'Traditional'
      }
    }
  )

  i ++
}

// reset counter for memorized version
// start step count, timer
i = 0
start = Now()

Push(
  out_dict['features'],
  {
    attributes: {
      step: i,
      duration: DateDiff(Now(), start),
      method: 'Memorized'
    }
  }
)

// states and counties
var memstates = Memorize(FeatureSetByPortalItem(
  Portal('https://arcgis.com'),
  '8c2d6d7df8fa4142b0a1211c8dd66903',
  0,
  ['STATE_FIPS', 'POPULATION'],
  false
))

var memcounties = Memorize(FeatureSetByPortalItem(
  Portal('https://arcgis.com'),
  '3c164274a80748dda926a046525da610',
  0,
  ['NAME', 'STATE_FIPS', 'POPULATION'],
  false
))

// loop through counties
for (var c in memcounties) {
  // get parent state feature
  var the_state = First(Filter(memstates, `STATE_FIPS = '${c['STATE_FIPS']}'`))

  Push(
    out_dict['features'],
    {
      attributes: {
        step: i,
        duration: DateDiff(Now(), start),
        method: 'Memorized'
      }
    }
  )

  i ++
}


return FeatureSet(Text(out_dict))

I won’t go through this whole expression. What I’m doing here is getting a featureset of A) US States, and B) US Counties.

For each feature in the Counties FeatureSet, I filter the States FeatureSet, grab that feature, and calculate a derived statistic. I’m not actually keeping that stat, though.

Every fifty counties, the duration of the process is recorded. The process is then repeated, but first memorizing the FeatureSets.

Histograms!

Yeah, Dashboards have bar charts.

But do they have binning?

The Process

1. Calculate min and max
2. Determine size of each bin
3. For each bin:
   1. Filter data in bin range
   2. Count number of records
   3. Create a feature with bin number, count

The Function

function Histogram(fset, field, bins) {
  var hmin = Min(fset, field)
  var hmax = Max(fset, field)
  var step = Floor((hmax - hmin) / (bins - 1))

  var hist_fs = {
    fields: [
      {name: 'bin_num', type: 'esriFieldTypeInteger'},
      {name: 'bin_name', type: 'esriFieldTypeString'},
      {name: 'the_count', type: 'esriFieldTypeInteger'}
    ],
    geometryType: '',
    features: [{
      attributes: {
        bin_num: 0,
        bin_name: `< ${Floor(hmin + step)}`,
        the_count: Count(Filter(fset, `${field} < ${hmin + step}`))
      }
    }]
  }

  var b_idx = 1
  
  for (var i = hmin + step; i < (hmax - step); i += step) {
    var the_count = Count(Filter(fset, `${field} > ${i} AND ${field} < ${i + step}`));
      Push(
          hist_fs['features'],
          { attributes: {
              bin_num: b_idx,
              bin_name: `${Floor(i)} — ${Floor(i + (step - 1))}`,
              the_count: the_count
          }}
      )
      
      b_idx += 1
  }

  Push(hist_fs['features'],
  { attributes: {
      bin_num: b_idx,
      bin_name: `> ${Floor(i)}`,
      the_count: Count(Filter(fset, `${field} > ${i}`))
    }
  })

  return FeatureSet(Text(hist_fs))
}

Dynamic Binning?

What if the bin number could change?

What if it could be dynamic? Is that even possible?

Sort of!

All the Bins

What we can do is nest the binning portion of the function and add an incrementing field for the bin_set. With this many count and filter functions going on, memorize is going to come in really handy.

When the bins are all together, the result is sort of a meta-distribution.

But let’s apply a numeric filter to it and give our dashboard a slider! Now how’s it look?

Table Widgets

Colors and Hex Codes

Messing with numbers is easier than with hex codes

Arcade function ToHex() is a one-way street

hexed

Given an array of RGB values, this function returns the appropriate hex code:

function hexed(color_array) {
  return `#${Concatenate(Map(color_array, ToHex))}`
}

hexed([255,150,56]) → "#ff9638"

Striping

Row striping is built in.

But not when you enable Advanced Formatting!

darken

With numeric color arrays, darkening a color is as simple as:

function darken(val) { return val - 15 }

Put it together: striper

function striper(color_array) {
  return Iif(
    $rowindex % 2 == 1,
    hexed(color_array),
    hexed(Map(color_array, darken))
  )
}

Tables ♥ HTML

Enough of the big functions. Let’s look at some smaller ideas.

You can pass all kinds of HTML into a table, and it will render it.

SVG Icons

Aren’t there built-in icons?

• Only aligns left or right
• Fixed size
• Awkward to add, name

Literal SVG Code

Remember to set height and fill!

var alert_svg = Decode(
  $datapoint.alert,
  'GREEN', `<svg height=40px viewBox="0 0 24 24"><title>alert-circle</title><path fill="green" d="M13,13H11V7H13M13,17H11V15H13M12,2A10,10 0 0,0 2,12A10,10 0 0,0 12,22A10,10 0 0,0 22,12A10,10 0 0,0 12,2Z" /></svg>`,
  'YELLOW', `<svg height=40px viewBox="0 0 24 24"><title>alert-rhombus</title><path fill="gold" d="M12 2C11.5 2 11 2.19 10.59 2.59L2.59 10.59C1.8 11.37 1.8 12.63 2.59 13.41L10.59 21.41C11.37 22.2 12.63 22.2 13.41 21.41L21.41 13.41C22.2 12.63 22.2 11.37 21.41 10.59L13.41 2.59C13 2.19 12.5 2 12 2M11 7H13V13H11V7M11 15H13V17H11V15Z" /></svg>`,
  'ORANGE', `<svg height=40px viewBox="0 0 24 24"><title>alert</title><path fill="orange" d="M13 14H11V9H13M13 18H11V16H13M1 21H23L12 2L1 21Z" /></svg>`,
  'RED', `<svg height=40px viewBox="0 0 24 24"><title>alert-decagram</title><path fill=red" d="M23,12L20.56,9.22L20.9,5.54L17.29,4.72L15.4,1.54L12,3L8.6,1.54L6.71,4.72L3.1,5.53L3.44,9.21L1,12L3.44,14.78L3.1,18.47L6.71,19.29L8.6,22.47L12,21L15.4,22.46L17.29,19.28L20.9,18.46L20.56,14.78L23,12M13,17H11V15H13V17M13,13H11V7H13V13Z" /></svg>`,
  ''
)

return {
  cells: {
    alert: {
      displayText: alert_svg,
      textAlign: 'center'
    }
  }
}

HTML “Chart” with Inline Style

• Make a <div> and use style="…"
• Hard-code the “max” value

var mag_div = `<div style="width:${$datapoint.mag / 8 * 100}%; background-color:blue; color:white; padding:4px; font-weight:bold">
  ${Text($datapoint.mag, '#.00')}
</div>`

Dynamic Buttons

var btn_style = `padding:5px; background-color:#333; font-weight:bold; color:white; line-height:30px; border-radius:7px; text-align:center;`

var action_button = Decode(
  $datapoint.alert,
  'RED', `<a style="${btn_style}" href="tel:1234567890">That's serious! Call Someone!</a>`,
  'ORANGE', `<a style="${btn_style}" href="mailto:someone@email.com">Important. Send an e-mail</a>`,
  'YELLOW', '',
  'GREEN', `<a style="${btn_style}" href="https://arcgis.com">Click to Close</a>`,
  ''
)

Long Strings, Multiple Lines

Long text in a list → variable height!

Long text in a table → truncated text or horizontal scrolling!

function Breakup(str, limit) {
  if (IsEmpty(limit)) { return 'Error! Specify a limit or you will crash your browser' }
  var out_arr = []
  var search_str = str

  While(Count(search_str) > limit){
    var nearest_break = Find(' ', search_str, limit - 6)
    Push(out_arr, Left(search_str, nearest_break))
    search_str = Trim(Right(search_str, Count(search_str) - (nearest_break)))
  }

  Push(out_arr, search_str)

  return Concatenate(out_arr, '<br>')
}

See, wouldn’t it be nice if we could get our table to automatically adjust height for a maximum width?

Until recently, you had to hard-code the number of breaks using nested if statements. There wasn’t a great way to break up a long string at desired intervals.

At 1.19, Arcade introduced While loops, so now we can just keep breaking the string up until our remainder is small enough.

Could you go the other way? Build the string word by word and inserting line breaks as you exceed the limit? Yes! But I didn’t! I wanted a good reason to test out the While loop.

Going the Other Way

function Buildup(str, limit) {
  var out_str = ''
  var words = Split(str, ' ')
  var line_len = 0

  for (var w in words) {
    out_str += words[w]
    line_len += Count(words[w])
    if (line_len > limit) {
      out_str += '<br>'
      line_len = 0
    } else {
      out_str += ' '
    }
  }

  return Trim(out_str)
}

Honestly, this is a lot safer. Plus, it can be done with an older version of Arcade!

While loops are cool, but it’s easy to get it stuck if you’re not certain that you will eventually meet the condition.

Goofy Stuff

Notes Table as Chat Convo

var prtl = Portal('https://maps.co.kendall.il.us/portal')

var fs = FeatureSetByPortalItem(
  prtl,
  'da490f45ce954edca8ba4a5cd156564b',
  1,
  ['remarks', 'doc_guid', 'created_user', 'created_date']
)

var out_dict = {
  fields: Splice(Schema(fs)['fields'], [{name:'is_you', type:'esriFieldTypeString'}]),
  geometryType: '',
  features: []
}

for (var f in fs){
  Push(
    out_dict['features'],
    {
      attributes: {
        remarks: f['remarks'],
        doc_guid: f['doc_guid'],
        created_user: f['created_user'],
        created_date: f['created_date'],
        is_you: Iif(f['created_user'] == GetUser()['username'], 'yes', 'no')
      }
    }
  )
}

return FeatureSet(out_dict)

It’s silly, yes. But it demonstrates one very interesting idea. When we are creating a Data Expression, we have access to the GetUser function.

When we have the active user’s name, we can alter the script output with it. Ideas:

• A quality-control dashboard that only lets you see / review other people’s work.
• A staff dashboard that only exposes the active user’s data
• Load a different Survey123 form based on the user’s group
• Load a completely different layer for different users

Aggregate by Month, Split by Category

Packing and Unpacking JSON

var prtl = Portal('https://maps.co.kendall.il.us/portal')

function Memorize(fs) {
  var temp_dict = {
    fields: Schema(fs)['fields'],
    geometryType: '',
    features: []
  }

  for (var f in fs) {
    var attrs = {}

    for (var attr in f) {
      attrs[attr] = Iif(TypeOf(f[attr]) == 'Date', Number(f[attr]), f[attr])
    }

    Push(
      temp_dict['features'],
      {attributes: attrs}
    )
  }

  return FeatureSet(Text(temp_dict))
}

var fs = Memorize(Filter(FeatureSetByPortalItem(
  prtl,
  'da490f45ce954edca8ba4a5cd156564b',
  1,
  ['remarks', 'objectid', 'created_user', 'created_date']
), "created_date is not null and created_user <> 'kendall_gis'"))

var out_dict = {
  fields: [
    {name: 'theYear', type: 'esriFieldTypeInteger'},
    {name: 'theMonth', type: 'esriFieldTypeInteger'},
    {name: 'contents', type: 'esriFieldTypeString'}
  ],
  geometryType: '',
  features: []
}

// get distinct year/month combos

var dated = GroupBy(
  fs,
  ['objectid', 'remarks', 'created_user'],
  [
    {name: 'theYear', expression: 'extract(year from created_date)', statistic: 'sum'},
    {name: 'theMonth', expression: 'extract(month from created_date)', statistic: 'sum'}
  ]
)

var month_set = Distinct(dated, ['theYear', 'theMonth'])

for (var m in month_set) {
  var month_remarks = Filter(dated, `theYear = ${m['theYear']} and theMonth = ${m['theMonth']}`)

  var remark_json = {}

  for (var mr in month_remarks) {
    if (HasKey(remark_json, mr['created_user'])) {
      remark_json[mr['created_user']] += `;${mr['remarks']}`
    } else {
      remark_json[mr['created_user']] = mr['remarks']
    }
  }

  Push(
    out_dict['features'],
    { attributes: {
      theYear: m['theYear'],
      theMonth: m['theMonth'],
      contents: Text(remark_json)
    }}
  )
}

return FeatureSet(out_dict)

As written, the output FeatureSet is going to have just a few features, but the contents field will be massive.

List Item Template

var output = `<div style="display:flex">
<div style="width:15px; margin:5px; writing-mode:vertical-rl; font-size:larger; font-weight:bold;">
${mname}<br>
${$datapoint.theYear}
</div>`

for (var r in remarks) {

  var remarks_list = `<ul><li>${Concatenate(Split(remarks[r], ';'), '</li><li>')}</li></ul>`

  var the_div = `<div style="padding:3px; margin:3px; border-radius:5px; width:75%;">
  <h4 style="text-decoration:underline">${r}</h4>${remarks_list}
  </div>`

  output += the_div
}

We used to use something like this as a “work report” where staff could list off anything they worked on in a given day, and then get a month-by-month overview.

The sample dataset I used doesn’t make much sense for this format, but the idea could be applied to lots of things.

The general idea is that we can write our Data Expression in a way that leverages how the Dashboard works.

Since we can use FromJSON in the List’s Advanced Formatting, we can essentially pass

SVG Clock

Still one of my favorite useless things.

var n = $datapoint.eventTime
var thehour = Hour(n)
var theminute = Minute(n)

if (thehour > 12){
    thehour -= 12
}

var hourdegrees = thehour * 30 + (5 / theminute)
var minutedegrees = theminute * 6
var ampm = Iif(Hour(n) > 12, 'PM', 'AM')

Line Item Template

<p style="text-align:center;">
    <svg height="300" width="300">
      <circle cx="150" cy="150" fill="white" r="130" stroke="black" stroke-width="5"></circle>
      <path d="M150 150 L 150 40 Z" stroke="black" stroke-linecap="square" stroke-width="3" text-align:center="" transform="rotate({expression/minutedegrees},150,150)"></path>
      <path d="M150 150 L 150 60 Z" stroke="black" stroke-linecap="square" stroke-width="5" transform="rotate({expression/hourdegrees},150,150)"></path>
    </svg>
</p>

It’s not really useless, though it is a bit silly.

But again, think beyond the example you’re seeing. We can use feature attributes to create dynamic SVGs!

You do have to know a bit about how SVG works, but if you do, the options are pretty wide open. Different shapes, colors, styles.

If you’re working with geometry that has COGO attributes, like a legal description, you can actually have the SVG mirror the features themselves!

Sorry, no example on that one.

In Closing: General Tips

Backtick Strings (Template Literals)

'my name is ' + $feature.name + ', how do you do?'

is the same as

`my name is ${$feature.name}, how do you do?`

Backticks and Mixed Types

'my number is ' + Text($feature.id) + ', how do you do?'

is the same as

`my number is ${$feature.id}, how do you do?`

Backticks and Line Breaks

'we want a long string' + TextFormatting.NewLine +
'and we are willing' + TextFormatting.NewLine + 
'to work for it'

is the same as

`we want a long string
and we are willing
to work for it`

Please, use backticks. No more concatenating strings, no more casting as strings, no more NewLine things.

Just write the text you want and pass in your attributes as needed.

Use SQL Functions

GroupBy and Distinct

• Pass SQL commands to server
• Let the server do all the work
• Returns a FeatureSet for you, no cobbling together a dictionary
• Use the ObjectID to get the same features but with new fields

It’s not really “grouping” or “distinct” to use a unique field, but that’s still valid!

In your other function parameters, you can pass whatever sort of SQL you want, within reason.

Consider Data Expressions

• Faster than standard layers when done right
  • Limit incoming fields
  • Omit geometry if not needed
• Flexible output

When you add a layer to your dashboard, by default, you get everything.

But if you know you only want a few attributes and maybe don’t need the geometry, why waste the network traffic?

Write a data expression that cherry-picks the attributes you need.

Use the Console Command

• Pass things to Console('some message')
• Great for debugging an expression
• Messages get sent to your browser console, too!

Whitespace is Free

Use line breaks and spaces

No bonus points for conserving code lines

No trophies for Maximum Terseness, sorry

var clear_variable = Iif(
  i_can_understand == True,
  'this is good',
  'this is not good'
)

var tidy_dict = {
  one:   1,
  two:   2,
  three: 3,
  four:  4
}

You could write both of these as one-liners, but your readability will suffer. Don’t do it. A high line count in your expression is not necessarily a bad thing.

Comment!

You may not be the only person to look at your code.

You may forget why you wrote it that way two years ago.

Leave comments

/* Here is a longer explanation of
    the next section of code. I figure
    you might need the explanation. */

for (var n in m) {
  // and here's where we do the next thing
  IncomprehensibleFunction(some_variable)
}

// Future Me, you're welcome!

Questions?

Be in Touch!

josh@jcarlson.page

https://jcarlson.page

@jcarlson@mapstodon.space