Multivar imshow - #30597

I fixed the circleci doc error for this.
It would be great if someone could take a look :)
@QuLogic @story645 @ksunden @timhoffm

General thoughts on return types:
Doing things like float | tuple[float, ...] as is done for several things here (vmin/vmax, clip, etc) is potentially problematic.
Humans may easily work with that, but type checkers will likely yell that they didn't check for all possible outcomes

We discussed change the behaviour of colorizer to always return tuples on the call last week.

The relevant moving parts here are:

1. Norm (Normalize, MultiNorm): members: vmin, vmax, clip
2. Colorizer: members: get/set_clim, get/set_clip, vmin, vmax, clip
3. _ColorizingInterface: members: get/set_clim, get/set_clip

The Norm ABC must be typed as follows for backwards compatibility:
def vmin(self) -> float | tuple[float | None, ...] | None: ...

For the Colorizer, I think it makes sense to force tuples on the getter, but allow both on the setter:

    def get_clim(self) -> tuple[tuple[float | None, ...], tuple[float | None, ...]]: ...
    def set_clim(self, vmin: float | tuple[float, ...] | None = ..., vmax: float | tuple[float, ...] | None = ...) -> None: ...

For the _ColorizingInterface we have two options.
A: allow both
def set_clim(self, vmin: float | tuple[float, float] | tuple[float | None, ...] | None, vmax: float | tuple[float | None, ...] | None = ...) -> None: ...
B: Allow get/set_clim only when using scalar data, and otherwise encourage the user to use the colorizer interface:
def set_clim(self, vmin: float | tuple[float, float] | None = ..., vmax: float | None = ...) -> None: ...

    def get_clim(self):
        """
        Return the values (min, max) that are mapped to the colormap limits.

        This function is not available for multivariate data.
        """
        if self._colorizer.norm.n_components > 1:
            raise AttributeError("`.get_clim()` is unavailable when using a colormap "
                                 "with multiple components. Use "
                                 "`.colorizer.get_clim()` instead.")
        return self.colorizer.norm.vmin, self.colorizer.norm.vmax

One reason why I favor option B, is that set_clim is already sufficiently complicated, because for scalar data it allows both signatures:
.set_clim(vmin=vmin, vmax=vmax)
.set_clim((vmin, vmax))
and the 2nd option is ambiguous if there are two colors

@ksunden @story645 Could you let me know what you think?

@timhoffm @ksunden
As discussed at the meeting on Thursday, I have now rebased this PR.
Once this is merged, we can more easily review #31214

@timhoffm
I am putting my replies here so that I can resolve the comments above without hiding the answers for future reference.

What about structured arrays, are they supported too? Do we possibly need one place to define "multivariate data" that can be referenced?

Yes we support structured data as well. On a related note the internal _ImageBase class requires structured data, and as requested I updated the docstring of this so that it reads:

            - a (M, N) array interpreted as scalar (greyscale) image,
              with one of the dtypes `~numpy.float32`, `~numpy.float64`,
              `~numpy.float128`, `~numpy.uint16` or `~numpy.uint8`.
            - a (M, N) structured array with K fields for multivariate colormapping.
              This must be used with a `.BivarColormap` (K=2) or generally with a
              K-component `.MultivarColormap`.
            - (M, N, 4) RGBA image with a dtype of `~numpy.float32`,
              `~numpy.float64`, `~numpy.float128`, or `~numpy.uint8`.

How should we add the option of structured data to the top level functions?
Should we include the two ways to include multivariate data on one line, i.e. something like this?:

            - a (K, M, N) scalar array or a structured (M, N) array with K fields:
              a K-component M*N mesh for multivariate colormapping. This must be 
              used with a `.BivarColormap` (K=2) or generally with a K-component 
              `.MultivarColormap`.

Also, did we discuss (K, M, N) vs. (M, N, K)? (Sorry in case I bring up topics that we may have discussed before)

Yes, this has been discussed, and it this keeps coming back up. I believe the primary discussions on this was at the weekly meeting around this time. I know I have made multiple posts on this before, but I have a difficult time finding them among all the different PRs.
If we want to discuss this again I suggest we bring it up at a weekly meeting and make sure that this time we write some things in the meeting notes :)

Also, it states "This parameter is ignored if X is RGB(A)."

I'm changing this to Scalar colormaps are ignored if *X* is RGB(A). which better reflects the current behaviour both on main and in this pr.

plt.imshow(np.random.random((6, 7, 3), cmap='not_a_colormap')

will cause an exception on main, while only if the cmap argument is a valid scalar colormap is the paramater ignored, i.e.:

plt.imshow(np.random.random((6, 7, 3), cmap='viridis')

How do we know RGB(A), i.e. shape (M, N, 3) or (M, N, 4) if there is (K, N, M) as multivariate data. Is this now a heuristic that the first or last dimension is low?

The multivariate pipeline is triggered by a valid multivariate colormap, thus we have:

i.e.:

plt.imshow(np.random.random((3, 3, 3))                        →   interpreted as rgb image
plt.imshow(np.random.random((3, 3, 3), cmap='viridis')        →   interpreted as rgb image
plt.imshow(np.random.random((3, 3, 3), cmap='not_a_cmap')     →   raises an error
plt.imshow(np.random.random((3, 3, 3), cmap='3VarAddA')       →   uses a multivariate colormap

@timhoffm A follow up on this would be much appreciated :)

Mostly small changes that I noticed when re-reading the PR. Let's still get them in. Please add them as an additional commit for easier review. We'll squash-merge in the end.

@timhoffm I really need you to finish this review/approve this so that we can get it merged and other people can start reviewing #31214

Also, did we discuss (K, M, N) vs. (M, N, K)? (Sorry in case I bring up topics that we may have discussed before)

Yes, this has been discussed, and it this keeps coming back up. I believe the primary discussions on this was at the weekly meeting around #28428 (comment) time. I know I have made multiple posts on this before, but I have a difficult time finding them among all the different PRs.
If we want to discuss this again I suggest we bring it up at a weekly meeting and make sure that this time we write some things in the meeting notes :)

Of the top of my head, I see the primary motivation in that (K, M, N) is the array shape of a list of component arrays: [component_1, ..., component_K]. Though that's not too compelling. There is precedence for handling "list of datasets" differently that 2D array of datasets, e.g. in boxplot() or in grouped_bar.

For RGB, the shape would be (M, N, 3), and I'm inclined to draw an analogy from the color channels to components. This would speak for (M, N, K).

But the more important argument would be: What is the typical data structure people already have when they are working with multivariate data? - And I'm completely blank here. Do you have insights?

The multivariate pipeline is triggered by a valid multivariate colormap, thus we have:

i.e.:

plt.imshow(np.random.random((3, 3, 3))                        →   interpreted as rgb image
plt.imshow(np.random.random((3, 3, 3), cmap='viridis')        →   interpreted as rgb image
plt.imshow(np.random.random((3, 3, 3), cmap='not_a_cmap')     →   raises an error
plt.imshow(np.random.random((3, 3, 3), cmap='3VarAddA')       →   uses a multivariate colormap

This also means, if you have multivariate data, you must always specify a colormap. Is this documented explicitly? If not please add it because that's important to know.

It's probably ok to request this as there would need to be different multivar colormaps depending on the data. OTOH it may be nice to be able to drop some multivar data and matplotlib figures out a nice visualization. This could be added in the future; i.e. infer multivar or not primarily from the data structure; in case of ambiguit, e.g. (3, 3, 3), check the colormap, and if none was given, fall back to a scalar interpretation. This would be a straight forward extension of the current logic.

Should we include the two ways to include multivariate data on one line, i.e. something like this?:

            - a (K, M, N) scalar array or a structured (M, N) array with K fields:
              a K-component M*N mesh for multivariate colormapping. This must be 
              used with a `.BivarColormap` (K=2) or generally with a K-component 
              `.MultivarColormap`.

Yes.

@timhoffm Thank you for coming back to this :)

This also means, if you have multivariate data, you must always specify a colormap. Is this documented explicitly? If not please add it because that's important to know.

This is a good point. I was thinking the docstring could get this across, the I do not think I was sufficiently clear with the language. This should be better:

            - a (K, M, N) scalar array or a structured (M, N) array with K fields:
              a K-component M*N mesh for multivariate colormapping. A valid
              `.BivarColormap` (K=2) or K-component `.MultivarColormap` must be
              specified using the *cmap* keyword argument.

NOTE: I also change the section that describes the array in the docstring of pcolor so that it matches the docstring of pcolormesh. Both support RGB image data, but this was undocumented in the docstring for pcolor.

We have the option to detect when the user might be trying to use this feature in colors._ensure_multivariate_data(), where we could raise an error with a hint, for example if the user attempts to input data of shape (k, n, m) with 2 <= k < 10 and n,m >10 without the required colormap. Currently such data will raise an error, but the error raised by 'imshow' is different from the error raised by 'pcolor', 'pcolormesh'. We don't want to change these errors, as there are other top level functions that can invoke those errors that do not support multivariate color mapping (yet and perhaps never). I think we can get the feature in first, and then see if we want to update the errors later.

But the more important argument would be: What is the typical data structure people already have when they are working with multivariate data? - And I'm completely blank here. Do you have insights?

I can mostly speak from my own experience, and I have had use for this feature in two contexts:

1. I am plotting results from two separate experiments together – i.e. the data is loaded from two different sources.
2. I am plotting the results from two separate data processing pipelines acting on the same dataset.

In both cases, I end up with two separate handles, data_A, data_B, and the call signature ax.imshow((data_A, data_B), ...) is the most convenient.

(I started implementing multivariate color mapping in matplotlib because I used to work with dark-field X-ray microscopy which typically produces 4D datasets that reduce to a series of 2D datasets through moment analysis (intensity, center of mass in x[orientation], center of mass in theta [strain], 2nd moments, ...), and we needed 2D colormaps to visualize them. Note that we would often plot things that are qualitatively very different together, such as orientation[mrad] and strain[dimensionless], in order to look for correlations. We would use different computational pipelines to get to the different moments, and represent them with different variable names.)

I have limited experience with fluorescence microscopy, but I think it is worth noting that the typical format OMETIFF stores data as (T, C, Z, Y, X), where T is typically time, and C is typically the color [fluorophore? wavelength?]. Both T, C, and Z follow before Y and X because the microscope always only reads 2D images sequentially, and this becomes the natural way to store the data as it comes out of the instruement.

The only argument I have seen for (N, M, K), is the analogy to RGB images, however to me this is a very weak argument, as the two approaches are trying to achieve very different things. You should not visualize data from an RGB camera using multivariate color mapping, and likewise, you should not map some arbitrary data space to RGB – doing so is likely to cause you to misinterpret your data. With this in mind, have different call signatures for the two approaches is to me a feature to help avoid mistakes.

This needs to pass PR cleanliness but otherwise looks good to me too. @trygve are you waiting on input from anybody else?

This needs to pass PR cleanliness but otherwise looks good to me too. @trygvrad are you waiting on input from anybody else?

I think this can be squash merged now, and that should take care of the cleanliness :)

@trygvrad thanks for the thorough discussions and for going through all the review rounds with us! This has been a lot of effort, but I'm convinced we're on a good path for consistently adding multivariate plotting so that it will be intuitive and easy to use within matplotlib. 🚀

Thank you @timhoffm.
The next step is that I rebase #31214 . I will probably get to it next weekend :)