This class represents a rewrite pattern list that has been frozen, and thus immutable. This replaces the uses of OwningRewritePatternList in pattern driver related API, such as dialect conversion. When PDL becomes more prevalent, this API will allow for optimizing a set of patterns once without the need to do this per run of a pass.

Differential Revision: https://reviews.llvm.org/D89104

There are several pieces of pattern rewriting infra in IR/ that really shouldn't be there. This revision moves those pieces to a better location such that they are easier to evolve in the future(e.g. with PDL). More concretely this revision does the following:

* Create a Transforms/GreedyPatternRewriteDriver.h and move the apply*andFold methods there.
The definitions for these methods are already in Transforms/ so it doesn't make sense for the declarations to be in IR.

* Create a new lib/Rewrite library and move PatternApplicator there.
This new library will be focused on applying rewrites, and will also include compiling rewrites with PDL.

Differential Revision: https://reviews.llvm.org/D89103

Adds support for
- Dropping unit dimension loops for indexed_generic ops.
- Folding consecutive folding (or expanding) reshapes when the result
  (or src) is a scalar.
- Fixes to indexed_generic -> generic fusion when zero-dim tensors are
  involved.

Differential Revision: https://reviews.llvm.org/D90118

The alignment attribute in the 'alloca' op treats the '0' value as 'unset'.
When parsing the custom form of the 'alloca' op, ignore the alignment attribute
with if its value is '0' instead of actually creating it and producing a
slightly different textually yet equivalent semantically form in the output.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D90179

Based on discourse discussion, fix the doc string and remove examples with
wrong semantic. Also fix insert_map semantic by adding missing operand for
vector we are inserting into.

Differential Revision: https://reviews.llvm.org/D89563

This revision allows the fusion of the producer of input tensors in the consumer under a tiling transformation (which produces subtensors).
Many pieces are still missing (e.g. support init_tensors, better refactor LinalgStructuredOp interface support, try to merge implementations and reuse code) but this still allows getting started.

The greedy pass itself is just for testing purposes and will be extracted in a separate test pass.

Differential revision: https://reviews.llvm.org/D89491

This has been deprecated for >1month now and removal was announced in:

https://llvm.discourse.group/t/rfc-revamp-dialect-registration/1559/11

Differential Revision: https://reviews.llvm.org/D86356

This reverts commit b22e2e4c.

Investigating broken builds

This has been deprecated for >1month now and removal was announced in:

https://llvm.discourse.group/t/rfc-revamp-dialect-registration/1559/11

Differential Revision: https://reviews.llvm.org/D86356

Add folder for the case where ExtractStridedSliceOp source comes from a chain
of InsertStridedSliceOp. Also add a folder for the trivial case where the
ExtractStridedSliceOp is a no-op.

Differential Revision: https://reviews.llvm.org/D89850

Differential Revision: https://reviews.llvm.org/D89853

Reuse most code for printing/parsing/verification from SubViewOp.

https://llvm.discourse.group/t/rfc-standard-memref-cast-ops/1454/15

Differential Revision: https://https://reviews.llvm.org/D89720

https://llvm.discourse.group/t/rfc-standard-memref-cast-ops/1454/15

Differential Revision: https://reviews.llvm.org/D89784

[mlir] Simplify DDR matching patterns with equal operands for operators where it's applicable. Added documentation.

This https://reviews.llvm.org/D89254 diff introduced implicit matching between same name operands.

Differential Revision: https://reviews.llvm.org/D89598

MLIRTransforms is needed to provide BufferizeTypeConverter
definitions.

A "structural" type conversion is one where the underlying ops are
completely agnostic to the actual types involved and simply need to update
their types. An example of this is shape.assuming -- the shape.assuming op
and the corresponding shape.assuming_yield op need to update their types
accordingly to the TypeConverter, but otherwise don't care what type
conversions are happening.

Also, the previous conversion code would not correctly materialize
conversions for the shape.assuming_yield op. This should have caused a
verification failure, but shape.assuming's verifier wasn't calling
RegionBranchOpInterface::verifyTypes (which for reasons can't be called
automatically as part of the trait verification, and requires being
called manually). This patch also adds that verification.

Differential Revision: https://reviews.llvm.org/D89833

A "structural" type conversion is one where the underlying ops are
completely agnostic to the actual types involved and simply need to update
their types. An example of this is scf.if -- the scf.if op and the
corresponding scf.yield ops need to update their types accordingly to the
TypeConverter, but otherwise don't care what type conversions are happening.

To test the structural type conversions, it is convenient to define a
bufferize pass for a dialect, which exercises them nicely.

Differential Revision: https://reviews.llvm.org/D89757

Reviewed By: herhut

Differential Revision: https://reviews.llvm.org/D89262

Differential Revision: https://reviews.llvm.org/D89825

Without this I get a warning about not all paths returning.

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D89760

It's unfortunate that this requires adding a dependency on scf dialect
to std bufferization (and hence all of std transforms). This is a bit
perilous. We might want a lib/Transforms/Bufferize/ with a separate
bufferization library per dialect?

Differential Revision: https://reviews.llvm.org/D89667

Add bufferizations for extract_element and tensor_from_elements.

Differential Revision: https://reviews.llvm.org/D89594

AllReduceLowering is currently the only GPU rewrite pattern, but more are coming. This is a preparation change.

Reviewed By: herhut

Differential Revision: https://reviews.llvm.org/D89370

```
   ...
   cond_br %cond, ^bb1(...), ^bb2(...)
 ...
 ^bb1: // has single predecessor
   ...
   cond_br %cond, ^bb3(...), ^bb4(...)
```

 ->

```
   ...
   cond_br %cond, ^bb1(...), ^bb2(...)
 ...
 ^bb1: // has single predecessor
   ...
   br ^bb3(...)
```

Differential Revision: https://reviews.llvm.org/D89604

The initial goal of this interface is to fix the current problems with verifying symbol user operations, but can extend beyond that in the future. The current problems with the verification of symbol uses are:
* Extremely inefficient:
Most current symbol users perform the symbol lookup using the slow O(N) string compare methods, which can lead to extremely long verification times in large modules.
* Invalid/break the constraints of verification pass
If the symbol reference is not-flat(and even if it is flat in some cases) a verifier for an operation is not permitted to touch the referenced operation because it may be in the process of being mutated by a different thread within the pass manager.

The new SymbolUserOpInterface exposes a method `verifySymbolUses` that will be invoked from the parent symbol table to allow for verifying the constraints of any referenced symbols. This method is passed a `SymbolTableCollection` to allow for O(1) lookups of any necessary symbol operation.

Differential Revision: https://reviews.llvm.org/D89512

Adding unroll support for transfer read and transfer write operation. This
allows to pick the ideal size for the memory access for a given target.

Differential Revision: https://reviews.llvm.org/D89289

The opposite of tensor_to_memref is tensor_load.

- Add some basic tensor_load/tensor_to_memref folding.
- Add source/target materializations to BufferizeTypeConverter.
- Add an example std bufferization pattern/pass that shows how the
  materialiations work together (more std bufferization patterns to come
  in subsequent commits).
  - In coming commits, I'll document how to write composable
  bufferization passes/patterns and update the other in-tree
  bufferization passes to match this convention. The populate* functions
  will of course continue to be exposed for power users.

The naming on tensor_load/tensor_to_memref and their pretty forms are
not very intuitive. I'm open to any suggestions here. One key
observation is that the memref type must always be the one specified in
the pretty form, since the tensor type can be inferred from the memref
type but not vice-versa.

With this, I've been able to replace all my custom bufferization type
converters in npcomp with BufferizeTypeConverter!

Part of the plan discussed in:
https://llvm.discourse.group/t/what-is-the-strategy-for-tensor-memref-conversion-bufferization/1938/17

Differential Revision: https://reviews.llvm.org/D89437

Parsing of a scalar subview did not create the required static_offsets attribute.
This also adds support for folding scalar subviews away.

Differential Revision: https://reviews.llvm.org/D89467

Each hardware that supports SPV_C_CooperativeMatrixNV has a list of
configurations that are supported natively. Add an attribute to
specify the configurations supported to the `spv.target_env`.

Reviewed By: antiagainst, ThomasRaoux

Differential Revision: https://reviews.llvm.org/D89364

The current fusion on tensors fuses reshape ops with generic ops by
linearizing the indexing maps of the fused tensor in the generic
op. This has some limitations
- It only works for static shapes
- The resulting indexing map has a linearization that would be
  potentially prevent fusion later on (for ex. tile + fuse).

Instead, try to fuse the reshape consumer (producer) with generic op
producer (consumer) by expanding the dimensionality of the generic op
when the reshape is expanding (folding).  This approach conflicts with
the linearization approach. The expansion method is used instead of
the linearization method.

Further refactoring that changes the fusion on tensors to be a
collection of patterns.

Differential Revision: https://reviews.llvm.org/D89002

Part of the refactor discussed in:
https://llvm.discourse.group/t/what-is-the-strategy-for-tensor-memref-conversion-bufferization/1938/17

Differential Revision: https://reviews.llvm.org/D89271

Once we have tensor_to_memref ops suitable for type materializations,
this pass can be split into a generic type conversion pattern.

Part of the refactor discussed in:
https://llvm.discourse.group/t/what-is-the-strategy-for-tensor-memref-conversion-bufferization/1938/17

Differential Revision: https://reviews.llvm.org/D89258

Part of the refactor discussed in:
https://llvm.discourse.group/t/what-is-the-strategy-for-tensor-memref-conversion-bufferization/1938/17

Differential Revision: https://reviews.llvm.org/D89261

Differential Revision: https://reviews.llvm.org/D87684

This revision adds a programmable codegen strategy from linalg based on staged rewrite patterns. Testing is exercised on a simple linalg.matmul op.

Differential Revision: https://reviews.llvm.org/D89374

Update linalg-to-loops lowering for pooling operations to perform
padding of the input when specified by the corresponding attribute.

Reviewed By: hanchung

Differential Revision: https://reviews.llvm.org/D88911

The build of MLIR occasionally fails (especially on Windows) because there is missing dependency between MLIRLLVMIR and MLIROpenMPOpsIncGen.

1) LLVMDialect.cpp includes LLVMDialect.h
2) LLVMDialect.h includes OpenMPDialect.h
3) OpenMPDialect.h includes OpenMPOpsDialect.h.inc, OpenMPOpsEnums.h.inc and OpenMPOps.h.inc

The OpenMP .inc files are generated by MLIROpenMPOpsIncGen, so MLIRLLVMIR which builds LLVMDialect.cpp should depend on MLIROpenMPOpsIncGen

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D89275

TensorConstantOp bufferization currently uses the vector dialect to store constant data into memory.
Due to natural vector size and alignment properties, this is problematic with n>1-D vectors whose most minor dimension is not naturally aligned.

Instead, this revision linearizes the constant and introduces a linalg.reshape to go back to the desired shape.

Still this is still to be considered a workaround and a better longer term solution will probably involve `llvm.global`.

Differential Revision: https://reviews.llvm.org/D89311

This combines two separate ops (D88972: `gpu.create_token`, D89043: `gpu.host_wait`) into one.

I do after all like the idea of combining the two ops, because it matches exactly the pattern we are
going to have in the other gpu ops that will implement the AsyncOpInterface (launch_func, copies, alloc):

If the op is async, we return a !gpu.async.token. Otherwise, we synchronize with the host and don't return a token.

The use cases for `gpu.wait async` and `gpu.wait` are further apart than those of e.g. `gpu.h2d async` and `gpu.h2d`,
but I like the consistent meaning of the `async` keyword in GPU ops.

Reviewed By: herhut

Differential Revision: https://reviews.llvm.org/D89160