Friday, June 26, 2015

A short history of Salesforce bugs and issues

Below is a history of the Salesforce bugs/issues that I've either directly raised with Salesforce or been involved in. While not comprehensive at the moment, it is mostly useful for me to have a single place the references them all.

  1. Bulk API gets error "MISSING_ARGUMENT:fiedXXX__c not specified:--" when using "N/A#"
    A customer using the Scribe data loading tool encountered a MISSING_ARGUMENT error when trying to create OpportunityLineItem records where there is a custom field from our managed package. The custom field is an external id, but not required.
    It turned out to be a limitation in Scribe - Topic: "Upsert issue - issue with multiple External Ids?". It can't handle multiple External Id fields and just uses the first one encountered for the upsert, even if you aren't populating it with data.
  2. Events continue to get delivered to Streaming API client which uses a session id , the session which has been explicitly terminated The Streaming API continues to send messages for expired/deleted Sessions - Case 11745520
  3. Spring '15 - Compile error occurs when deploying a test class using an @isTest(SeeAllData=true) method with a separate class using @TestSetup method
  4. "Generate WSDL" generates a WSDL that does not contain the definition of the compound types address and location if API version is 30.0 or above
  5. WSDL-Based Asynchronous Callouts using Continuations can't be unit tested.
  6. The Tooling API isn't enforcing the package version dependency in the class/trigger metadata
  7. Tooling API SymbolTable contains “private” visibility modifiers instead of “protected”
  8. The SOAP Tooling API is missing the SessionHeader in WSDL operation
  9. Querying ApexExecutionOverlayResult via the Tooling API gives an UNKNOWN_EXCEPTION

Friday, June 12, 2015

Using the Salesforce Metadata API to run a single test method in isolation

In my previous post The Holy Grail of Testing in Salesforce I covered my journey towards being able to execute a single Apex test method in isolation of the others defined in an Apex class. Here I'll cover how it actually works.

The Metadata API

In working with the Metadata API deploy() method as part of a continuous integration projection it dawned on me that I already had everything I needed to run a single test method in isolation.

As long as the test class and test method in question were public I could create a temporary wrapper test class that calls just the target test method.

For example, say the target was the Foo ApexClass and method bar(). Something like this, but with more actual testing and assertions going on:

@IsTest
public class Foo {
    public static testMethod void bar() {
        System.Debug(LoggingLevel.Debug,'Example Foo.bar() method');
        // Assertions and stuff
    }
}

Then to invoke this via a Metadata API deploy call all I need to do is create a wrapper class, the associated .cls-meta.xml, and package.xml to send in the zip payload. The wrapper class can be as simple as:

@IsTest
public class FooWrapper {
    public static testMethod void barWrapper() {
        Foo.bar();
    }
}

The most important part in deploy call is the DeployOptions.

  • Setting checkOnly to true will prevent any changes being actually deployed to the org. The FooWrapper class only exists to execute the test.
  • runTests(string[]) allows the deployment to run a specific test case. This should be set to the name of the wrapper test class that is being deployed. E.g. "FooWrapper"
    Once v34.0 is widely available the testLevel should also be set to RunSpecifiedTests.

The DebuggingHeader can be used to control the debug log contents that comes back in the response.

After starting the deploy process it is just a matter of monitoring the AsyncResult and DeployResult until it complete and then extracting the RunTestsResult.

The end result

Given an Apex testing class and the name of a test method it contains I can run a checkOnly deployment that will run just that test. As I mentioned before, the test in question needs to be public so that the wrapper class can access it. There are also likely to be issues if the @testSetup annotation is being used in the target class.

The future

It occured to me that this would also be a great mechanism for running anonymous apex in a testing context as well. The anonymous apex could easily be wrapped in a temporary class annotated with @IsTest and a test method. It could optionally specify @IsTest(SeeAllData=true) as well.

Additional test methods could be created in the wrapper class to target other test methods.


Update: Tooling API runTestsAsynchronous

Nate Lipke kindly pointed out to me that the Summer '15 release includes a runTestsAsynchronous method that accepts the target class Id's and a testMethods parameter "containing the name of a test method in the test class."

The REST version takes this as a POST request. The SOAP API also has this method, but it isn't apparent how to specify the target test methods yet.

If you are using the Force.com IDE v34.0.0.20150511 Beta Version you can use the new API to run a specific test method.

From the Summer '15 - Tooling API Updates

POST JSON arrays of test methods to runTestsAsynchronous
You can now POST a JSON array of test methods to the runTestsAsynchronous endpoint. Formerly, in POST methods, therunTestsAsynchronous endpoint accepted only comma-separated lists of class IDs.

Format:
POST
/runTestsAsynchronous/
   Body:
   {"tests":}

Example <tests array>:

[{
  "classId" : "<classId 1>",
  "testMethods" : ["testMethod1","testMethod2","testMethod3"]
 },{
  "classId" : "<classId 2>",
  "testMethods" : ["testMethod1","testMethod2"]
}];

Thursday, June 11, 2015

FuseIT SFDC Explorer 2.7 Features

The new v2.7 release of the FuseIT SFDC Explorer has some new features.

Run a single test method in isolation

After running tests, either in synchronous or asynchronous mode, you can right click on in individual test method to get the context menu. This will give you the option to run just that test method in isolation. See The Holy Grail of Testing in Salesforce.

Basic Apex Class editor

There is the new Apex Classes tab that has search and editor. It's no where are full featured as something like Mavens Mate in Sublime, but it might be useful in a pinch if the developer console is misbehaving (or you want to bypass certain permission restrictions).

Highlighting code coverage lines

For larger Apex classes, where it can be difficult to find the largest blocks of code that need coverage, I've added coverage markings to the scroll bar.

It's still a bit of a work in progress getting everything aligned correctly, but it should make it easier to scroll to the blocks of lines without coverage.

Alpha version of an Org Metadata diff tool

Here you establish a secondary Salesforce Session. Then, using the diff button the tool will pull down the metadata for both orgs. Once stored in the file system it is passed off the the Diff tool of your choosing.

Miscellaneous

There are some general enhancements to Wsdl2Apex to deal with WSDL's that have multiple imports of the same namespace. Support for 2 character pod identifiers. And some generally improved error logging.

The Holy Grail of Apex Testing in Salesforce

The quest has been long and difficult, but I think I've finally found it. The Holy Grail of testing Apex in Salesforce. (Skip directly to the end result)

Many would consider the introduction of the @isTest annotation in the Spring 12 release to be one of the most important releases for Apex testing. Or even the subsequent ability to create Price Book Entries when using @isTest(SeeAllData=true) in Summer 14. Freeing the majority of remaining cases that still needed to access live data. There has also been the shift from synchronous test execution to asynchronous test execution.

Those are indeed note worthy, but I think I've found something more useful.

Origins

It seems like a long time ago, but the idea exchange tells me it was "about" 2 years ago (in the frustrating way it refuses to show actual specific dates on anything). It's like there is some psychology at play where not having the specific date doesn't make it feel as old. I digress.

I had the idea that if anonymous apex could be run in a testing context then it could be used to run individual test methods from the suite of tests that an Apex Class contains. Feel free to take a minute and vote for it, I'll wait - Run anonymous apex as if it were a test case.

The end game was to be able to run a single test method of interest without having to manipulate the Apex class to do it. Sure you can comment out the @IsTest annotations or testMethod keywords on the tests that aren't of interest, but that is just an extra step towards yak shaving and away from what you want to be doing. Running a single test method, checking the test result and inspecting the resulting debug log.

There would also be other side benefits to having anonymous apex code able to run in a testing context. Isolation from live data, testing out code before actually using it, callout mocks...

As with most good ideas that are worth having once, someone else had also had the idea. Perhaps more succinctly towards the actual goal. Allow single test method execution from an Apex Class. Vote for that one too.

Dreamforce 2014

No great quest would be complete without a great journey, so in 2014 I voyaged 11,000 km (6800 miles) from Nelson, New Zealand to Dreamforce in San Francisco to pitch my idea to any Salesforce employee I could find. I also found time for a side quest of presenting at said conference.

The Meet the Developers session was a great place to raise the idea in front of those who might make it happen. I also managed to talk directly to Josh Kaplan about it in the halls.

The Product Managers that say Ni No!!

I demand a 2,500 voting point threshold. Ni!!

As you may have already read on my idea, it was largely hacked to bits with the suggestion to cause a rollback by throwing an error or using an assertion. True, in most cases this would rollback the transaction. It does however miss the main goal of running an individual test method. There are other limitations of it as well. For instance, callouts can't be rolled back.

More to the point, as per the response in the meet the developers session and the response in the other idea, there is a backlog item where Salesforce will add native support for running single test methods. Long term this makes more sense for the platform rather than hacking it in by using anonymous Apex.

Still. "You make me sad"

Not to be deterred by such a small flesh wound, I kept at the end game goal, looking for viable alternatives.

Then an accumulation of the work I'd been doing with various parts of Salesforce gave me an idea. It all came together to make something quite useful.

The Holy Hand Grenade

At this stage I'm not sure I want to expose how it works internally. A determined developer could probably root it out once they realize it is possible.
You can see details from how it works in Using the Salesforce Metadata API to run a single test method in isolation. It is also available in the FuseIT SFDC Explorer version 2.7 onward.

You can select an individual test method to run in isolation via a context menu on both the Apex Tests and Apex Test Queued tabs. Then you can see the results of that test and review the resulting Apex log. This will be done in isolation of any other test methods in the class. Be gone "MAXIMUM DEBUG LOG SIZE REACHED"!. No modifications to the Apex test class or org in geneneral are made as part of this process.

As with all such things, there are some requirements. The Apex class containing the test methods and the methods to be targeted both need to be public. Also, I haven't checked it with Apex classes using the @testSetup annotation, but I suspect it won't work. Static constructors should be OK.


UPDATE: Summer '15

So, it turns out in the Summer '15 (v34.0) of the Tooling REST API it is possible to POST JSON to the /runTestsAsynchronous/ that specifies both the Apex class Ids of the test classes and the test method names as well. This comes back to the feature being integrated into the platform overall.

It isn't currently apparent if the SOAP version of the Tooling API has the same capability. It does have the runTestsAsynchronous() method that takes a string parameter. The example in ApexTestQueueIte shows it being passed a comma seperated list of Apex Class Ids.

Further details

Tuesday, May 19, 2015

The mystery of the nondeterministic Salesforce test case

After making some changes to a trigger that can ultimately be fired after changes to an Opportunity are made I encountered an existing test case that started failing. The particular test case bulk inserts Opportunity and OpportunityLineItem records and ultimately goes on to bulk update OpportunityLineItem records that have revenue schedules.

What's odd is:

  • The test case specifically declares
    @IsTest(SeeAllData=false)
  • I checked and the test doesn't use instances of
    DateTime.now()
    or similar
  • Same for
    Math.random()

Due to the nature of the test case I thought the problem may have been around bulkification. As a starting point I dropped the counter down at the start of the test method that specifies how many Opportunity records to test against.

With only 2 opportunities being inserted the test case appeared to pass with every run.

Scaling it up to 50 opportunities resulted in the test case failing with almost every run.

Opportunity Count Run 1 Run 2 Run 3 Run 4 Run 5
2 Pass Pass Pass Pass Pass
28 Pass Fail Pass Fail Pass
30 Pass Fail Pass Pass Fail
31 Fail Pass Fail Pass Fail
50 Fail Pass Fail Fail Pass
62 Fail Fail Fail Fail Fail
150 Fail Fail Fail Fail Fail

This was odd, as I'd expect a test case that wasn't using random data to produce consistent results. What was more odd was that varying the number of Opportunities altered the pass/fail pattern. Generally, the more opportunities there were the greater the odds of the test failing.

The clue to the answer was in the test failure message, which in hindsight I should have focused on more than the bulkification.

System.DmlException: Insert failed. First exception on row 0; first error: CANNOT_INSERT_UPDATE_ACTIVATE_ENTITY, MyNamespace.FooOpportunityTrigger: execution of AfterInsert

caused by: System.DmlException: Upsert failed. First exception on row 25; first error: DUPLICATE_VALUE, duplicate value found: MyNamespace__OpportunityIdDynamicPropertyIdUnique__c duplicates value on record with id: a0E4000001GRmaU: []

Class.MyNamespace.FooOpportunityHelper.mapUdfFields: line 636, column 1
Class.MyNamespace.FooOpportunityHelper.handleAfter: line 515, column 1
Trigger.MyNamespace.FooOpportunityTrigger: line 15, column 1: []

The OpportunityIdDynamicPropertyIdUnique__c custom field is used on a custom object to make a composite key out of the Opportunity Id and another Id. It's primary purpose is to enforce uniqueness.

The field is populated using a workflow rule formula as follows:

Knowing the the issue is around the custom field that is assigned the opportunity id and that there are failures consistently with 62 records and 50% of the time with around 31 records indicated the problem was with the case sensitivity of the opportunity id in the field (See What are Salesforce ID's composed of?).

The Id's of the custom object weren't resetting with each test run. Instead the values were accumulating across each test run. For example, if the Opportunity Ids 0064000000fNiOM and 0064000000fNiOm appeared in the same test run it would fail immediately after the Worflow action completed.

The formula needed to use the case insensitive version of the Opportunity Id or the custom field uniqueness needed to be case sensitive. CASESAFEID() seemed like the easiest change as the case sensitivity couldn't be changed on an existing managed package field. It did create a migration problem for existing data.

Tuesday, April 21, 2015

FuseIT SFDC Explorer v2.5 features

The new v2.5 release of the FuseIT SFDC Explorer has some new features.

Reporting the rolling 24 hour API Requests

In addition to the existing tracking of API calls made for the duration of the session the rolling 24 hour API usage and limit will be reported:

Monitoring Asynchronous Test execution and results

When running the automated test cases I usually care first and foremost about any tests that are failing. If anything is failing the other tests that passed are just noise.

The developer console, as at Spring 15, gives a tree control that shows the test classes and the status of the individual methods. To view any individual errors you need to bring up the table view for the entire test run.

You can also view the same data under Setup > App Setup > Develop > Apex Test Execution.

I've taken elements of each of these views and reworked them a bit to highlight any failed tests. Anything that passed is hidden away under an initially collapsed node. Selecting an individual failed test will show both the error message and the stack trace. You can reselect individual test class and use them to start a new test run for just those classes.

List of Apex Classes with coverage

Several releases ago Salesforce removed the really useful Code Coverage column from the Apex Classes page. The idea to add it back in was rejected based on no having to maintain it in both places. See Ideas: Put back Code Coverage column to Apex Classes and Apex Triggers page

I found it to be a super useful column as you could sort the Apex Classes by size (Number of characters excluding comments) and then get an indication of which classes needed the most additional code coverage. I.e. find the biggest class by size with the lowest code coverage. Improving the coverage on this class will give you the biggest overall code coverage improvement.

Thursday, March 5, 2015

.NET Integration with Salesforce Tooling API Spring 15 (v33.0)

The Spring 15 Salesforce release brings some changes to the Tooling API. From the release notes:

Feature: Metadata namespace
Description: In previous versions of the SOAP Tooling API, metadata type elements were defined in the same namespace as the sObjects of the API, tooling.soap.sforce.com. In order to uniquely name these types, the suffix “Metadata” was appended to the metadata types. In this version of the API, a new namespace is introduced, metadata.tooling.soap.sforce.com, and the suffix is no longer used. The old names will continue to work with clients against older API endpoints.

You can see the change diffing the v32.0 WSDL with the newer v33.0 WSDL. The metadata ApexClass element from ApexClass is now also called ApexClass and resides in the namespace urn:metadata.tooling.soap.sforce.com (with the alias mns.

Having two complexTypes with the same name but different namespaces in the same WSDL creates a bit of a mess with Web References (WSDL.exe) and Service References (SvcUtil.exe).

With a web reference to the WSDL the ApexClass from the Metadata namespace gets the matching name and the ApexClass from the urn:tooling.soap.sforce.com namespace becomes ApexClass1. Not only does this break all the previous code that referenced ApexClass within the Tooling API but it is damn ugly.

    [System.CodeDom.Compiler.GeneratedCodeAttribute("System.Xml", "4.0.30319.34234")]
    [System.SerializableAttribute()]
    [System.Diagnostics.DebuggerStepThroughAttribute()]
    [System.ComponentModel.DesignerCategoryAttribute("code")]
    [System.Xml.Serialization.XmlTypeAttribute(Namespace="urn:metadata.tooling.soap.sforce.com")]
    public partial class ApexClass : Metadata {
    // ...
    }

    // ...
    /// 
    [System.CodeDom.Compiler.GeneratedCodeAttribute("System.Xml", "4.0.30319.34234")]
    [System.SerializableAttribute()]
    [System.Diagnostics.DebuggerStepThroughAttribute()]
    [System.ComponentModel.DesignerCategoryAttribute("code")]
    [System.Xml.Serialization.XmlTypeAttribute(TypeName="ApexClass", Namespace="urn:tooling.soap.sforce.com")]
    public partial class ApexClass1 : sObject {
        // ...
        private ApexClass metadataField; 
        // ...
        /// 
        [System.Xml.Serialization.XmlElementAttribute(IsNullable=true)]
        public ApexClass Metadata {
            get {
                return this.metadataField;
            }
            set {
                this.metadataField = value;
            }
        }
        // ...
    }

To be fair, this is an issue with the .NET generation of the classes in different namespaces as much as anything. The same WSDL in the FuseIT version of Wsdl2Apex creates separate Apex classes for each namespace to keep the class definitions distinct.

One brute force way to fix this is to rename the classes after the codegen is complete. Find the ApexClass that inherits from Metadata and rename it to ApexClassMetadata. Then rename ApexClass1 to ApexClass. Rinse and repeat for any other types that derive from Metadata (WorkflowRule1, WorkflowFieldUpdate1, ValidationRule1, Profile1, ...).

It might be possible to use the /namespace: option on SvcUtil.exe to move each WSDL namespace into a separate C# namespace. It doesn't appear that Wsdl.exe has the same option.