Monday, June 24, 2019

FuseIT SFDC Explorer 3.12.19175.1 - Summer '19

Another roundup of some of the changes to the FuseIT SFDC Explorer since the 3.11.19071.3 release.

Fix for Salesforce breaking the Partner API

One of the primary reasons to use the Partner API in an application is that it is org agnostic. It doesn't really care which org you are connecting to and will adjust to fit the shape of the org it is currently working with. So as long as you don't change the API version in the URL it will keep working indefinitely. Or at least, that is the theory...

Sadly, recent changes around the Real-Time Event Monitoring Beta caused metadata on a JSON field type to leak back into older Salesforce API versions. See Known Issue: DescribeSObject call using the v45.0 Partner API are failing due to a complex type(json) that isn't defined in the WSDL.

By updating to v46.0 of the Partner API we now avoid this issue. In older version it will manifest as a "There is an error in XML document" exception when making the describe call.

View Executed Units from an Apex Log

It's true, the Developer Console already has this information in a sub tab under the Execution Overview. Right now my version just shows the execution count for each Apex method and the basic execution time stats. It if isn't showing up, try turning on the timeline or parse tree first. The debug log will need to show the METHOD_ENTRY and METHOD_EXIT entries, so it the Apex Code logging level will need to be FINE or lower.

Wsdl2Apex : Make it easier to debug out stub objects

Currently, if you try and debug out any of the proxy sObjects that get created for a SOAP callout you end up with a ton of extra *_type_info arrays. While necessary for WebServiceCallout.invoke they have zero usage in debugging the state of the request. They are generally constant and don't really reflect the useful state details.

I'd previously written about using JSON serialization to make repeating a SOAP callout easier. Then it occured to me, why not make all the _type_info arrays transient. Then they won't even appear in the debug output. That's exactly what I've done with the current WSDL2Apex build.

Sadly, transient doesn't work for object.toString(). I did raise an idea for it - Provide a mechanism to exclude object variables/properties from toString()

Reciprocal Logging Event selection and highlighting

There are additional context menu options on a debug log to navigate between related opening and closing events. When applicable you can also select the parent event.

Other changes 3.11

  • Update all API calls to use v46.0 Summer '19
  • Support for monitoring multiple test jobs.
  • ApexTestRunner: New async methods for checking test status. Beta throttled test runner.
  • Fix bug where the 10th column in SOQL results was always being blanked out.
  • Data Export CLI: Allow for parameters to vary in order.
  • Expanded support for deploying different Metadata types.
  • MetadataServiceWrapper - Support for .layout metadata types. Improve creation of package zips - only create the required folders.
  • Option to extract login credentials from a URL formed with "un" and "pw" query string parameters.
  • Wsdl2Apex: Handle SimpleType parameters. Warn if a Union simpleType is encountered.
  • Fix serverURL construction from sfdx cli logins.
  • Handle exceptions with SFDX logins

Tuesday, April 9, 2019

Trailhead: Deep Learning and Natural Language Processing

Recently I tackled the Deep Learning and Natural Language Processing Trailhead module.

I can best summarize the experience with an image based on the Suppose you have one rabbit meme explanation of arithmetic:

To be fair, the first unit in the module does call out some prerequisites:

This is an advanced topic, and this module assumes you have a basic understanding of machine learning vocabulary, some experience with Python, and at least a little hands-on experience working with machine learning data and algorithms. If you don’t already have that background, you can get yourself up to speed using the following resources.

I personally think I meet those prerequisites. While I don't work with Python day to day the syntax is familiar enough I figured I can fake it till I make it.

The first two units were reasonably straight forward.

However, by the third unit - Apply Deep Learning to Natural Language Processing it started to get complicated. In particular, with the Hands-on Logistic regression questions 6, and to a lesser extent question 7. These both required completing several TODO lines of Python to derive the loss after 100 epochs.

Let's look at the code from the first part of that that needed to be completed in the TODO sections:

The challenge here is, as with all programming, that all the steps need to be completed successfully before you will get the expected answer. Get any of them wrong and things will go pear shaped fast. I could save you the pain of solving this challenge and provide you the full script, but that doesn't really go with the spirit on Trailhead. Instead I'll add some debugging outputs at various points to show the state of the tensors to hopefully make it clearer what should be happening. At least then if things start to go off track you can pick up the problem immediately.

Exercise 6

TODO: Generate 2 clusters of 100 2d vectors, each one distributed normally, using only two calls of randn()

println(classApoints)
tensor([[ 0.3374, -0.1778],
        [-0.3035, -0.5880],
        [ 0.3486,  0.6603],
        [-0.2196, -0.3792],
        #...
        [-0.7952, -0.9178],
        [ 0.4187, -1.1123],
        [ 1.1227,  0.2646],
        [-0.4698,  1.0866],
        [-0.8892,  0.7647]])
assert(classApoints.size() == torch.Size([100, 2]))
println(classBpoints)
tensor([[ 0.4771,  0.7203],
        [-0.0215,  1.0731],
        [-0.1408, -0.5394],
        [-1.2782, -0.8107],
        #...
        [ 1.1051, -0.5454],
        [ 0.1073,  0.8727],
        [-1.2800, -0.4619],
        [ 1.4342, -1.2103],
        [ 1.3834,  0.0324]])
assert(classBpoints.size() == torch.Size([100, 2]))

TODO: Add the vector [1.0,3.0] to the first cluster and [3.0,1.0] to the second.

println(classApoints)
tensor([[ 1.3374,  2.8222],
        [ 0.6965,  2.4120],
        [ 1.3486,  3.6603],
        [ 0.7804,  2.6208],
        #...
        [ 0.2048,  2.0822],
        [ 1.4187,  1.8877],
        [ 2.1227,  3.2646],
        [ 0.5302,  4.0866],
        [ 0.1108,  3.7647]])
println(classBpoints)
tensor([[ 3.4771,  1.7203],
        [ 2.9785,  2.0731],
        [ 2.8592,  0.4606],
        [ 1.7218,  0.1893],
        #...
        [ 4.1051,  0.4546],
        [ 3.1073,  1.8727],
        [ 1.7200,  0.5381],
        [ 4.4342, -0.2103],
        [ 4.3834,  1.0324]])

TODO: Concatenate these two clusters along dimension 0 so that the points distributed around [1.0, 3.0] all come first

println(inputs)
tensor([[ 1.3374,  2.8222],
        [ 0.6965,  2.4120],
        [ 1.3486,  3.6603],
        [ 0.7804,  2.6208],
        #...
        [ 4.1051,  0.4546],
        [ 3.1073,  1.8727],
        [ 1.7200,  0.5381],
        [ 4.4342, -0.2103],
        [ 4.3834,  1.0324]])

println(inputs.size())
torch.Size([200, 2])

TODO: Create a tensor of target values, 0 for points for the first cluster and # 1 for the points in the second cluster.

println(classA)
tensor([ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  0])

println(classA.size())
torch.Size([100])
println(classB)
tensor([ 1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
         1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
         1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
         1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
         1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
         1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
         1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
         1,  1])

println(classB.size())
torch.Size([100])

# TODO: Initialize a Linear layer to output scores for each class given the 2d examples

println(model)
Linear(in_features=2, out_features=2, bias=True)

# TODO: Define your loss function

Here you need to decide if you are going to use:

  1. MSE Loss for Regression
  2. NLLLoss for Classification
  3. CrossEntropyLoss for Classification

Worst case here, you could just try them each one at a time until you get an answer that matches the expected answers in the Trailhead challenge.

Finishing up exercise 6

After that the rest should fall into place fairly easily based on the prior examples.

Exercise 7: Logistic Regression with a Neural Network

Firstly, the instruction when I completed this included this:

# forward takes self and x as input # passes x through linear_to_hidden, then a tanh activation function, # and then hidden_to_linear and returns the output

I suspect that should actually be "and then hidden_to_output and returns the output".

The torch.tanh function is required for forward.

Initialize your new network to have in_size 2, hidden_size 6, and out_size 2

You are going to use the NeuralNet class that you just completed defining here.

Define your loss function

As with exercise 6, you can just try the 3 example loss functions to find a good fit for the expected answers.

Finishing up exercise 7

Most of the other parts for this question all fall into place based on the prior example.

Results

It would be accurate to say that this last unit took me way longer than the 90 minutes that Trailhead indicated. But dammit I earned those 100 measly points.

Monday, April 1, 2019

Trailhead Electric Imp plus plus

After completing the the Trailhead Electric Imp project I was left wanting an excuse to access a number of other features of the imp001 hardware. Namely the accelerometer and pressure sensor. Yet there didn't seem many opportunities for a static fridge to fully utilize them.

While I was experimenting with this back in May 2018 the #BeABuilder challenge happened and I got sidetracked using the hardware for that instead. Somewhere in that process I never got around to hitting publish on this blog post, so here we are a year or so latter with a better late than never post.

Firstly I expanded the data model in Salesforce so I'd have fields to receive the additional data. This included the acceleration and pressure readings.

Out of interest I tried getting the IoT Orchestration Traffic into a Lightning component. I didn't have much luck at the time and it didn't really liked to be iframed in.

Hunting through the Salesforce Labs I found the salesforce-iot-toolkit. This provided some great visualizations via platform events as the data came in from the fridge. It directly monitored the platform events to drive the graphs. I found I needed to use the source based version rather than the app exchange version, which was missing a number of features.

Dramatic reenactment of sensor usage

Modified Agent Nut Source

A few points of interest:

  1. My source was the version before they added proper refresh token support to handle expired sessions. So you probably don't want my version of getStoredCredentials()
  2. Added #require statements to access additional sensors.
  3. Dropped the READING_INTERVAL_SEC down to 1 second to improve the accelerometer data.
  4. Retrieved the X,Y, and Z axis acceleration values.
  5. Retrieved the pressure sensor and additional temperature reading from that sensor as well.
  6. Set the RGB color based on the current orientation indicated by the acceleration.
  7. The LIS2DH12 sensor needed a non-default I2C address of 0x32.

Modified Salesforce Platform Event Trigger Code

Nothing fancy here. Just collect the additional properties the map them to the new custom fields.

trigger SmartFridgeReadingReceived on Smart_Fridge_Reading__e (after insert) {
    List records = new List(); 
    for (Smart_Fridge_Reading__e event : Trigger.New) {
        SmartFridge__c record = new SmartFridge__c();
        record.deviceId__c = event.deviceId__c;
        record.temperature__c = event.temperature__c;
        record.humidity__c = event.humidity__c;
        record.door__c = event.door__c;
        record.ts__c = event.ts__c;
        record.accX__c = event.accX__c;
        record.accY__c = event.accY__c;
        record.accZ__c = event.accZ__c;
        record.pressure__c = event.pressure__c;
        record.temp2__c = event.temp2__c;
        records.add(record); }
    insert records;
}

Tuesday, March 12, 2019

FuseIT SFDC Explorer 3.11.19071.3 - Spring '19

Another roundup of some of the changes to the FuseIT SFDC Explorer since the 3.9.18190.1 release.

Bypass sObject types based of suffix to speed up metadata retrieval

With the update to v45.0 of the APIs there is now describe metadata coming back for change data capture with the suffix __ChangeEvent. This can be useful, except every custom object also gets a change event. If you have a large number of custom objects it can quickly become overwhelming.

I've added a filter option to the sObject metadata browser that defaults to only showing __c, __mdt, and __e. Other options can be turned on if you want to see them. A refresh is required for the changes to take affect.

Expanded Asynchronous Test Results view

After the async tests have finished running you will set the total duration by Apex class and method. This can be useful for identifying which test methods contribute the most to the overall test run time.

Expose the logging levels in the menu

You can alter the current logging levels for the current DebugLevel and then use "Update Current TraceFlag" to apply them to Salesforce.

The "Disable Logging" menu option will remove the current TraceFlag, which will stop further logging from occurring.

Wsdl2Apex

There have been a number of changes to the Wsdl2Apex processing to support more edge cases:

  1. Allow for schema imports via schema imports (nested imports). Includes checks for circular references.
  2. If the complex type can't be found in the current schema search the imported schema for it.
  3. If a SimpleType has a restriction that isn't primitive then search the underlying types for the primitive type that can be converted to Apex.
  4. Detect repeated imported schema target namespaces
  5. Use common code to create Apex class names from complex types.
  6. Add generated Apex comments to classes that extend a base type.
  7. Attempt to compile the XmlSchema to resolve groups so they can be automatically expanded.
  8. Include warning messages on web methods where the return type can't be resolved from the namespace.

Include Apex class name in code coverage breakdown

The Apex class name and test method name will be included with the class coverage breakdown details.

JWT Authentication for Salesforce sessions

It is now possible to use the OAuth 2.0 JWT Bearer Token flow to authenticate with Salesforce. I'll expand on the steps to do this shortly, which will include configuring the connected app and registering the X509 Certificate.

Other changes 3.11

  • Middle click to hide tab. Improve tab hiding.
  • Improve support for using the ToolingAPI in the EntityViewer.
  • If directly updating a nullable int in the SOQL results treat it as such (rather than a string)
  • Format a VF_PAGE_MESSAGE in the log timeline
  • Format display of ApexResult and HeapDump from the Tooling API SOQL Queries.
  • Allow for a CultureInfo.DateTimeFormat.TimeSeparator other than ":". Force the SOQL building code to use the InvarientCulture so the required format is applied.
  • Update core SalesforceSession to use v45.0 (Spring 19) APIs
  • For the Apex Log Stack, include the Soql Execute Count
  • Option to run the most recent async test cases again after a successful metadata deployment.
  • EntityViewer - Allow for ToolingAPI entities.
  • Event Log Viewer. Only show the first 4000 rows of data.
  • Log Timeline control. Show icons for Test.startTest() and Test.stopTest() if they are detected.
  • Handle failure to query for recent apex test runs.
  • When generating anonymous Apex to get the current Session ID include the SalesforceBaseUrl
  • Code Generation - If generating without custom fields, exclude picklist values containing "__". These typically represent relationships to custom sObjects
    Detect and skip duplicate picklist value entries.
    Fix the generated properties for *Picklist to set based on the enum value rather than the description attribute.
    Detect references to sObjects that aren't exposed via the metadata.
  • Regenerate sObjects (plus associated services and data sources) for API v44.0.
  • CodeGeneration - If configured, exclude relationships to custom fields.
  • Don't generate types for __ChangeEvent, __History, __Share
  • CodeGeneration - generate "extends" for Apex classes extending other classes.
  • Expand details for WebServiceCallout.invoke with comments.
  • ApexClass (for generation) - Track if the Apex classes extends a base class.
  • FieldService - When extracting child elements from a query result skip nodes that aren't XmlElements.
  • ToolingServiceWrapper - Add describeGlobalCached() that caches describeGlobal() to improve performance.
  • SalesforceSession - If an exception occurs with a WebRequest include the complete URL in the Exception.Data
  • Internal caching of the Tooling API DescribeGlobalResult Metadata
  • Auto-size columns in SOSL & single page SOQL results
  • Improve usage of direct Csv loading. Support bulk load for larger operations. Allow the primary sObject type to be defined.
  • Change the Apex log layout with the treeview between horizontal and vertical.
  • Define logging levels via common menu, Handle hitting the maximum debug log count
  • Capture SSL/TLS config issues and provide help link to dedicated help page.
  • When checking metadata deployment successes, highlight changed components vs. unchanged.

Wednesday, October 3, 2018

Dreamforce 2018 Roundup / Summary

The recap for this Dreamforce is going to come across as a bit of a photo journal as I go back through my phones timeline to piece back together what I got up to. Somehow, even on my 5th time attending this conference things still went past at a frantic pace.

Table of Contents

  1. Preconference
  2. Day One
    • My Presentation
    • Main Keynote
  3. Day Two
    • The Lightning Platform Roadmap
    • Dreamfest
  4. Day Three
    • Developer Keynote
    • Meet the Developers
    • True to the Core
  5. Day Four
    • Mulesoft
  6. Deployment Fish

Preconference

A visit to the Salesforce Tower for the Tooling Partner meeting. Then a quick visit to the Salesforce Park (the day before it was closed due to structural damage).

Day One

Testing Lightning Flow Automations

  • Usage of Test.loadData() to prep records for testing from a static resource.
  • Starting auto launched flows from a testing context.
  • In Winter'19 - Use the FlowTestCoverage and FlowElementTestCoverage from the Tooling API to get coverage details for flows. See Track Process and Flow Test Coverage

DX Super Session (Video)

New scratch org "snapshots" to use as the starting point for creating additional scratch orgs. The snapshot can be used to pre-configure the scratch org in a known state. E.g. with the dependent packages installed and test data loaded. They are currently using the 0TT Trailforce Template keyprefix. Note that any orgPreferences are additive over what is already present in the snapshot.

spinners

FORCE_SHOW_SPINNER and FORCE_SPINNER_DELAY environment options that can make automating the CLI easier. They prevent the spinner coming back on the standard output.

Naming for unlocked package versions.

My Presentation

I've done a whole separate blog post to cover my talk - Dreamforce 2018 Presentation - Understand your Org shape via visualization of Metadata Component Dependencies.

Main keynote

Demo's of Einstein voice to convert speech, to text, to actions within Salesforce.

Customer 360 - Create a canonical view of the customer across Marketing Cloud, Commerce Cloud, and Service Cloud.

MVP Party

This year it was at the Tonga Room & Hurricane Bar. The band on the boat was πŸ‘Œ.

Day Two

The Lightning Platform Roadmap - Part I and Part II

Day Three

Developer Keynote

Lightning Developer Pro Sandboxes - Spin up faster than a traditional Sandbox. Approx 5 minutes to activate (versus a few hours to even days).

Another example of a named package version.

True to the Core

Lots of coverage of how Salesforce will work towards reinvigorating the idea exchange to handle the current scale. In particular, limiting each users ability to vote of every possible idea. Forcing them to focus on what is most important to them. Salesforce will then commit to working on the highest rated ideas.

Everything that's Awesome with Apex

(Slides)

Roadmap for Spring '19 and Summer '19

  • Exposing the Org Limit details in Apex. E.g. Daily Asysnc Apex executions.
  • Support for creating Scratch Orgs with access to Platform Cache.
  • Support for "rename symbol" refactoring in Apex via the Apex Language Services, and by extension Visual Studio Code. Longer term, this may expand to implementing interfaces and extracting variables.
  • Improved support for enforcing Field Level Security (FLS) and Create,Read,Update,Delete (CRUD) in Apex. Goal is to reduce the amount of processing that was historically required to do this. It's particularly important for those publishing on the App Exchange. https://twitter.com/FishOfPrey/status/1044978778684841984

Quip Party

The August Hall was a great venue. Including a bowling alley and mini arcade downstairs.

Day 4

MuleSoft


Deployment Fish Bottle Openers

To catch up on

See Also

Wednesday, September 26, 2018

Dreamforce 2018 Presentation - Understand your Org shape via visualization of Metadata Component Dependencies

At Dreamforce this year I gave a theater presentation on using the Metadata Component Dependency Pilot API along with Gephi to explore ways of untangling the "Happy Soup" that makes up a Salesforce org.

In this post I'll cover the contents of that talk and expanded on some areas that I'd like to cover in more depth.

Session recording

Why Visualize an Orgs Metadata?

All four sets are identical when examined using simple
summary statistics, but vary considerably when graphed - Source

If you've ever seen the Anscombe's quartet you'd know a compelling reason to visualize information in addition to straight out numerical analysis. Visualizations can reveal patterns and features in the data that otherwise might not be readily apparent.

Additionally, when we start drilling into all the metadata in an org and the relationships within that metadata we can very quickly be overwhelmed by the share volume of it. There can be thousands of metadata items and many more relationships between them.

The Soup and the Polar Bear

The concept of the metadata in your production org existing as a "Happy Soup" was introduced at TrailheadDX 2017 by Wade Wegner. It describes the scenario where all the conceptual apps that make up your production org intermingle without any strongly defined boundaries between them. Essentially, the org becomes one big melting pot of metadata. This has been the status quo for some time now (if you ignore managed packages and the relatively new addition of Packaging 2.0).

Soup chef at work on your production org - a Marvelous Metadata Mixing Pot

Now, take one of the production orgs that I work in day-to-day as an example. It was forged 11 years ago when the seasonal release logo was a polar bear and the API version was v8.0. Ever since then it's been accumulating metadata changes as multiple people have come and gone. Each person leaving a distinct mark on the metadata that makes up that org.

Back to our soup metaphor, that org is a decade old soup. Different soup chefs have been mixing ingredients in via a pinch of declarative UI changes, a dash of changes sets, a smidgen of metadata API deploys, and maybe the odd unmanaged package thrown in for good measure. Over the years an org can seem less like a carefully conceived soup recipe and more like a marvelous concoction with everything thrown in and reacting together in weird and wonderful ways.

Which kind of leaves you wondering - If you were presented with just the resulting happy soup, how would you identify the ingredients that went into it and how they need to interact so you could replicate it again? How can we unmake the happy soup?

The goal of our metadata visualizations will be to use the minds ability to quickly perceive and derive meaning from details such as size, color, position, and proximity to find insights that might not be apparent in tabular data. That might sound complicated, but the objective is to make otherwise complicated relationships readily apparent.

“By visualizing information, we turn it into a landscape that you can explore with your eyes. A sort of information map. And when you’re lost in information, an information map is kind of useful.” - David McCandless

So, we know we want to make visualizations of the metadata and the dependencies within, but where do we start gathering that data?

The Metadata Component Dependency API

Luckily for us the new Metadata Dependencies pilot introduced in Summer '18 at TrailheadDX allows for the use of Tooling API SOQL queries to quickly find dependencies between metadata components. You can signup for the pilot via your AE as per the release notes: Untangle Your Dependencies with MetadataComponentDependency Queries (Pilot)

The benefit of this pilot is that it is much faster and easier than trying to assemble the same information yourself. I've tried extracting similar details via the ApexClassMember.SymbolTable. While possible, it is a path that is thwart with multiple API calls per ApexClass. And at the end of all that you will still only have a fraction of the possible relationships covered.

Once you are in the pilot it is pretty simple to use. A single SOQL query can bring back the majority of relationships for an org. For example, using the Tooling API via the sfdx cli:

sfdx force:data:soql:query --usetoolingapi --query "SELECT MetadataComponentId, MetadataComponentName, MetadataComponentType, RefMetadataComponentId, RefMetadataComponentName, RefMetadataComponentType FROM MetadataComponentDependency Where RefMetadataComponentType = 'CustomField'" -u devorg

Supported Dependencies

As this is a pilot feature the types of dependencies it can reveal isn't 100% yet. Below is a small subset of what can and can't be tracked: (as at the time of posting)

TypeUsed IdSupported
CustomObjectApex Class
CustomObjectProcess Builder
CustomObjectLookup Relationship Field
CustomObjectReports
CustomFieldLayout
CustomFieldReports
FlowProcess Builder
FlowFlow

You can see the complete list of supported metadata relationships in https://github.com/afawcett/dependencies-sample

Augmenting the Graph

Using the results from the Metadata Component Dependency query as a starting point we can then augment the resulting graph with additional metrics. Here are some of the additional details that can be merged in as meta-metadata.

  • CustomField.Metadata - can be used to show lookup relationships
  • ApexClass.ApiVersion - useful for finding Apex classes with mismatched API versions
  • The code coverage results for each Apex class
  • The number of lines of code in each Apex Class
  • Where available, the created date for the Metadata
  • The number of records in a CustomObject
  • The number of Triggers per sObject

Tooling to create the Dependency Graph

The first part of the visualization process will be to query the MetadataComponentDependency records and convert the results into a graph. I didn't go into this step in any detail in my talk as it isn't very interesting to watch. At it's simplest, you can use one of the following options.

CLI Generation of the Gephi graph

NOTE: At the time of publishing the Github repo wasn't fully up to date with the latest public build. I'll aim to have this done shortly after Dreamforce.

Run the following command:

sfdx fitdx:dependencies:report -u DeveloperOrg > DeveloperOrg.gexf

And that's pretty much it. The command from the FitCLI plugin will run the query, build the graph, and export it ready for opening in Gephi. You only need to point it at the correct Salesforce Org and redirect the output into a file.

If you want to add the augmented graph metrics you can supply the -a option. Note that this will take significantly longer to complete as it involves a number of additional tooling API queries. It's also beneficial to run all the Apex Tests before exporting the graph so it has complete coverage details.

GUI option for creating the Gephi file

As an alternative to using the command line you can also using the FuseIT SFDC Explorer from v3.10 onwards. This now includes the Metadata Dependencies tab to generate Gephi file. It's pretty much exactly the same as the command line version and will prompt for where you want to save the resulting file.

Ideally with either the CLI or the GUI options the target org will have access to MetadataComponentDependency records. If that isn't the case you can still use the augmented option to get some graph details. It won't be as fast or as complete, but you will get a good deal of the relationships based on other data exposed via the Tooling API.

Examples

Now we can get into the interesting part of loading the graphs into Gephi to see what we can discover. You can open the .gexf files directly into Gephi and then start manipulating them.

This video quickly goes through the Field Usage and Packages examples that are expanded on below.

Field Usage

Lets say we want to find what will be impacted if changes are made to the Property__c.Address__c field in the Dreamhouse app. The steps would be as follows:

  1. Partition the Node color by Type. This will make it more apparent what type of nodes you are looking at.
  2. Size the nodes based on the Betweenness Centrality. Which is a fancy way of saying "make the nodes that appear on the shortest paths between other nodes bigger". This is one possible indication of how important a node is to all the other nodes in the graph. Before we can do this we need to run the Network Diameter statistics.
  3. Apply a layout to arrange the nodes. I've used the "Force Atlas" layout with the Repulsion strength set to 1500 and Adjust by Sizes checked.
  4. Show the Node Labels and scale to fit
  5. Switch to the Data Table window and filter to the Address field.
  6. Zoom out a bit, and you will be able to see the the Metadata that references the Property__c.Address__c field.

Potential Packages

  1. Run the "Modularity" Statistics. This algorithm will look for metadata communities based on the dependencies between them. You can adjust the Resolution up or down if you want more or less communities.
  2. Change the appearance of the Nodes to be partitioned by the resulting "Modularity Class". With the correct resolution and layout it will given you a reasonable guide to where potential packages are located. You will likely need to make some finer adjustments along the boundaries.
  3. One way to export the nodes for the package is to select them and then copy to a new workspace. The new data table can then be exported as a csv.

API Versions

  1. Filter to just those bits of metadata that have an API version defined. One way to do this is to drag a "Non-null (ApiVersion)" filter down onto the Queries.
  2. Under Appearance > Nodes, set the node size to rank by Degree. The degree is a measure of how many edges go in or out of the node. So more connected nodes will be larger.
  3. Set the Nodes color to be ranked by API version. Choose a color pallete that emphasizes lower API versions as being problematic.
  4. Set the Node Label to "API Version" and size as required.

Code Coverage

  1. Size the nodes by "LengthWithoutComments" to give an indication of how much code is in each Apex class.
  2. Color the nodes by ranking of the "PercentCovered". Use the slider to roughly indicate code with 75% coverage.
  3. It can be useful to filter to just the test classes and assign them a different node color. They won't have any coverage themselves.
  4. Set the node label to the "PercentCovered"
  5. A circular layout sorted by the coverage percent can be useful to separate out the problem nodes. Move the text classes into a seperate ring so you can see the relationships between the tests and the classes they cover.

Classes with excessive lines of code

As an experiment I drastically increased the amount of node scaling that was possible. This can really emphasis the differences in the size of Apex classes.

Friday, August 24, 2018

Dreamforce 2018 Session picks

Here are some of my current picks for Dreamforce 2018 sessions. I'm aiming for a mix of developer related topics in areas I want to learn more about plus anything that sounds informative. It isn't an exhaustive list and there are certainly some other session that I'll be adding.

Most important session that I'm definitely going to attend

I might be biased, but this session is on my must attend list.

Tell your friends, tell your neighbors, tell your family, bring your mum along. There will be something for everyone!
Who doesn't like pretty pictures, metadata, and a sprinkling of graph theory?

Keynotes

Meet The *'s

Apex

  • Everything that's Awesome with Apex
    Get a sneak peak into Apex plans, roadmap and how we are making secure development easier. We'll share some of our newest concepts (async Apex anyone?) and get your feedback on hot IdeaExchange features.
    πŸ‘€Async Apex, maybe something to help with FLS and CRUD in managed packages. Yes please!
  • Test Your Visual Workflow and Process Builder Automations. I'm assuming there will be some Apex involved in this testing process.

APIs

Salesforce DX

IDE

Platform Events

Platform

Einstein / AI

Security

Fun

  • Flood the Trailblazer Community Cove: Swag & Sticker Meetup
    Got too much swag or something you can't get home - swap it! Last year I gave a bunch of stuff away that I'd collected but couldn't actually put in my luggage. I'm looking at you oversized Codey. And the rose seed in soil that NZ customs would have opinions on.

Lightning

  • TODO - Need something to fill out this area to be a well rounded developer. Lightning Roadmap maybe?

See also