The lit test suite uses python 3.6 features. Rather than a strange
python syntax error upon running the lit tests, we will require the
correct version in CMake.

Reviewed By: serge-sans-paille, yln

Differential Revision: https://reviews.llvm.org/D95635

This does not change the behavior directly: the tests only run when
`-DMLIR_INCLUDE_INTEGRATION_TESTS=ON` is configured. However running
`ninja check-mlir` will not run all the tests within a single
lit invocation. The previous behavior would wait for all the integration
tests to complete before starting to run the first regular test. The
test results were also reported separately. This change is unifying all
of this and allow concurrent execution of the integration tests with
regular non-regression and unit-tests.

Differential Revision: https://reviews.llvm.org/D97241

The CMake changes in 2aa1af9b to make it possible to build MLIR as a
standalone project unfortunately disabled all unit-tests from the
regular in-tree build.

Differential Revision: https://reviews.llvm.org/D96306

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D95978

Add the necessary bits to CMakeLists to make it possible to configure
MLIR against installed LLVM, and build it with minimal need for LLVM
source tree.  The latter is only necessary to run unittests, and if it
is missing then unittests are skipped with a warning.

This change includes the necessary changes to tests, in particular
adding some missing substitutions and defining missing variables
for lit.site.cfg.py substitution.

Reviewed By: stephenneuendorffer

Differential Revision: https://reviews.llvm.org/D85464



Co-authored-by: Isuru Fernando <isuruf@gmail.com>

There are builders that do not have python 3.6. Revert until this situation can be rectified

This reverts commit 0703b075.

Previously, CMake would find any version of Python3. However, the project
claims to require 3.6 or greater, and 3.6 features are being used.

Reviewed By: yln

Differential Revision: https://reviews.llvm.org/D95635

Use cross-compilation approach for `mlir-linalg-ods-gen` application
similar to TblGen tools.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D94598

The test process of the ir_array_attributes.py depends on numpy. This patch checks numpy in Python bindings configuration.
- Add NumPy in find_package as a required component to check numpy.
- If numpy is found, print the version and include directory.

Differential Revision: https://reviews.llvm.org/D92276

* Was causing auto-detect of pybind11 to fail on clean configure.

Differential Revision: https://reviews.llvm.org/D92043

* Makes `pip install pybind11` do the right thing with no further config.
* Since we now require a version of pybind11 greater than many LTS OS installs (>=2.6), a more convenient way to get a recent version is preferable.
* Also adds the version spec to find_package so it will skip older versions that may be lying around.
* Tested the full matrix of old system install, no system install, pip install and no pip install.

Differential Revision: https://reviews.llvm.org/D91903

In ODS, attributes of an operation can be provided as a part of the "arguments"
field, together with operands. Such attributes are accepted by the op builder
and have accessors generated.

Implement similar functionality for ODS-generated op-specific Python bindings:
the `__init__` method now accepts arguments together with operands, in the same
order as in the ODS `arguments` field; the instance properties are introduced
to OpView classes to access the attributes.

This initial implementation accepts and returns instances of the corresponding
attribute class, and not the underlying values since the mapping scheme of the
value types between C++, C and Python is not yet clear. Default-valued
attributes are not supported as that would require Python to be able to parse
C++ literals.

Since attributes in ODS are tightely related to the actual C++ type system,
provide a separate Tablegen file with the mapping between ODS storage type for
attributes (typically, the underlying C++ attribute class), and the
corresponding class name. So far, this might look unnecessary since all names
match exactly, but this is not necessarily the cases for non-standard,
out-of-tree attributes, which may also be placed in non-default namespaces or
Python modules. This also allows out-of-tree users to generate Python bindings
without having to modify the bindings generator itself. Storage type was
preferred over the Tablegen "def" of the attribute class because ODS
essentially encodes attribute _constraints_ rather than classes, e.g. there may
be many Tablegen "def"s in the ODS that correspond to the same attribute type
with additional constraints

The presence of the explicit mapping requires the change in the .td file
structure: instead of just calling the bindings generator directly on the main
ODS file of the dialect, it becomes necessary to create a new file that
includes the main ODS file of the dialect and provides the mapping for
attribute types. Arguably, this approach offers better separability of the
Python bindings in the build system as the main dialect no longer needs to know
that it is being processed by the bindings generator.

Reviewed By: stellaraccident

Differential Revision: https://reviews.llvm.org/D91542

This drops the use of deprecated CMake modules to find python.

Differential Revision: https://reviews.llvm.org/D91197

Introduce an ODS/Tablegen backend producing Op wrappers for Python bindings
based on the ODS operation definition. Usage:

  mlir-tblgen -gen-python-op-bindings -Iinclude <path/to/Ops.td> \
              -bind-dialect=<dialect-name>

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D90960

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D90955

[mlir] Change deprecated -Werror-implicit-function-declaration to -Werror=implicit-function-declaration

This -Werror- form has been deprecated since 2007 (GCC r122017).

And only enable it for CMAKE_C_FLAGS. C++ does not need it.

This is useful in C source files where it is easy for a typo to be
silently assumed by the compiler to be an implicit declaration.

Differential Revision: https://reviews.llvm.org/D90727

This patch introduces a SPIR-V runner. The aim is to run a gpu
kernel on a CPU via GPU -> SPIRV -> LLVM conversions. This is a first
prototype, so more features will be added in due time.

- Overview
The runner follows similar flow as the other runners in-tree. However,
having converted the kernel to SPIR-V, we encode the bind attributes of
global variables that represent kernel arguments. Then SPIR-V module is
converted to LLVM. On the host side, we emulate passing the data to device
by creating in main module globals with the same symbolic name as in kernel
module. These global variables are later linked with ones from the nested
module. We copy data from kernel arguments to globals, call the kernel
function from nested module and then copy the data back.

- Current state
At the moment, the runner is capable of running 2 modules, nested one in
another. The kernel module must contain exactly one kernel function. Also,
the runner supports rank 1 integer memref types as arguments (to be scaled).

- Enhancement of JitRunner and ExecutionEngine
To translate nested modules to LLVM IR, JitRunner and ExecutionEngine were
altered to take an optional (default to `nullptr`) function reference that
is a custom LLVM IR module builder. This allows to customize LLVM IR module
creation from MLIR modules.

Reviewed By: ftynse, mravishankar

Differential Revision: https://reviews.llvm.org/D86108

This reverts commit e9b87f43.

There are issues with macros generating macros without an obvious simple fix
so I'm going to revert this and try something different.

New projects (particularly out of tree) have a tendency to hijack the existing
llvm configuration options and build targets (add_llvm_library,
add_llvm_tool).  This can lead to some confusion.

1) When querying a configuration variable, do we care about how LLVM was
configured, or how these options were configured for the out of tree project?
2) LLVM has lots of defaults, which are easy to miss
(e.g. LLVM_BUILD_TOOLS=ON).  These options all need to be duplicated in the
CMakeLists.txt for the project.

In addition, with LLVM Incubators coming online, we need better ways for these
incubators to do things the "LLVM way" without alot of futzing.  Ideally, this
would happen in a way that eases importing into the LLVM monorepo when
projects mature.

This patch creates some generic infrastructure in llvm/cmake/modules and
refactors MLIR to use this infrastructure.  This should expand to include
add_xxx_library, which is by far the most complicated bit of building a
project correctly, since it has to deal with lots of shared library
configuration bits.  (MLIR currently hijacks the LLVM infrastructure for
building libMLIR.so, so this needs to get refactored anyway.)

Differential Revision: https://reviews.llvm.org/D85140

    Introduce an initial version of C API for MLIR core IR components: Value, Type,
    Attribute, Operation, Region, Block, Location. These APIs allow for both
    inspection and creation of the IR in the generic form and intended for wrapping
    in high-level library- and language-specific constructs. At this point, there
    is no stability guarantee provided for the API.

Reviewed By: stellaraccident, lattner

Differential Revision: https://reviews.llvm.org/D83310

Summary:
* Native '_mlir' extension module.
* Python mlir/__init__.py trampoline module.
* Lit test that checks a message.
* Uses some cmake configurations that have worked for me in the past but likely needs further elaboration.

Subscribers: mgorny, mehdi_amini, rriddle, jpienaar, shauheen, antiagainst, nicolasvasilache, arpith-jacob, mgester, lucyrfox, aartbik, liufengdb, stephenneuendorffer, Joonsoo, grosul1, Kayjukh, jurahul, msifontes

Tags: #mlir

Differential Revision: https://reviews.llvm.org/D83279

Summary:
Previous submit of new tests accidentally made this ON.
The tests should be opt-in.

To build with MLIR integration tests enabled, pass the following

cmake .... \
-DMLIR_INCLUDE_INTEGRATION_TESTS=ON \
....

Reviewers: mehdi_amini

Subscribers: mgorny, mehdi_amini, rriddle, jpienaar, shauheen, antiagainst, nicolasvasilache, arpith-jacob, mgester, lucyrfox, liufengdb, stephenneuendorffer, Joonsoo, grosul1, frgossen, Kayjukh, jurahul, msifontes

Tags: #mlir

Differential Revision: https://reviews.llvm.org/D81878

Summary:
This CL introduces an integration test directory for MLIR in general, with
vector dialect integration tests in particular as a first working suite. To
run all the integration tests (and currently just the vector suite):

$ cmake --build . --target check-mlir-integration
[0/1] Running the MLIR integration tests
Testing Time: 0.24s
Passed: 22

The general call is to contribute to this integration test directory with more
tests and other suites, running end-to-end examples that may be too heavy for
the regular test directory, but should be tested occasionally to verify the
health of MLIR.

Background discussion at:
https://llvm.discourse.group/t/vectorops-rfc-add-suite-of-integration-tests-for-vector-dialect-operations/1213/

Reviewers: nicolasvasilache, reidtatge, andydavis1, rriddle, ftynse, mehdi_amini, jpienaar, stephenneuendorffer

Reviewed By: nicolasvasilache, stephenneuendorffer

Subscribers: mgorny, mehdi_amini, rriddle, jpienaar, shauheen, antiagainst, nicolasvasilache, arpith-jacob, mgester, lucyrfox, liufengdb, stephenneuendorffer, Joonsoo, grosul1, frgossen, Kayjukh, jurahul, msifontes

Tags: #mlir

Differential Revision: https://reviews.llvm.org/D81626

Summary:
`mlir-rocm-runner` is introduced in this commit to execute GPU modules on ROCm
platform. A small wrapper to encapsulate ROCm's HIP runtime API is also inside
the commit.

Due to behavior of ROCm, raw pointers inside memrefs passed to `gpu.launch`
must be modified on the host side to properly capture the pointer values
addressable on the GPU.

LLVM MC is used to assemble AMD GCN ISA coming out from
`ConvertGPUKernelToBlobPass` to binary form, and LLD is used to produce a shared
ELF object which could be loaded by ROCm HIP runtime.

gfx900 is the default target be used right now, although it could be altered via
an option in `mlir-rocm-runner`. Future revisions may consider using ROCm Agent
Enumerator to detect the right target on the system.

Notice AMDGPU Code Object V2 is used in this revision. Future enhancements may
upgrade to AMDGPU Code Object V3.

Bitcode libraries in ROCm-Device-Libs, which implements math routines exposed in
`rocdl` dialect are not yet linked, and is left as a TODO in the logic.

Reviewers: herhut

Subscribers: mgorny, tpr, dexonsmith, mehdi_amini, rriddle, jpienaar, shauheen, antiagainst, nicolasvasilache, csigg, arpith-jacob, mgester, lucyrfox, aartbik, liufengdb, stephenneuendorffer, Joonsoo, grosul1, frgossen, Kayjukh, jurahul, llvm-commits

Tags: #mlir, #llvm

Differential Revision: https://reviews.llvm.org/D80676

Make ConvertKernelFuncToCubin pass to be generic:

- Rename to ConvertKernelFuncToBlob.
- Allow specifying triple, target chip, target features.
- Initializing LLVM backend is supplied by a callback function.
- Lowering process from MLIR module to LLVM module is via another callback.
- Change mlir-cuda-runner to adopt the revised pass.
- Add new tests for lowering to ROCm HSA code object (HSACO).
- Tests for CUDA and ROCm are kept in separate directories.

Differential Revision: https://reviews.llvm.org/D80142

This is equivalent to what is done for other projects (e.g. clang).

Differential Revision: https://reviews.llvm.org/D80022

In cmake, dependencies on generated files require some sophistication in the build system.  At build time, files are parsed to determine which headers they depend on and these dependencies are injected into the build system.  This works well with ninja, but has some constraints with the makefile generator.  According to the cmake documentation, this only works reliably within the same directory.

This patch expands the usage of mlir-headers to include all generated headers and adds an mlir-generic-headers target which triggers generation of dialect-independent headers.  These targets are used to express dependencies on generated headers.  This is mostly handled in AddMLIR.cmake and only a few CMakeLists.txt files need to change.

Differential Revision: https://reviews.llvm.org/D79242

Define MLIR_MAIN_INCLUDE_DIR, as it was not set anywhere.

Set MLIR_MAIN_SRC_DIR to the actual "source directory", and not the
"include directory" (as currently set).

Differential Revision: https://reviews.llvm.org/D77943

Setting MLIR_TABLEGEN_EXE would prevent building the native tool which is used in cross-compiling

Differential Revision: https://reviews.llvm.org/D75299

Putting this up mainly for discussion on
how this should be done. I am interested in MLIR from
the Julia side and we currently have a strong preference
to dynamically linking against the LLVM shared library,
and would like to have a MLIR shared library.

This patch adds a new cmake function add_mlir_library()
which accumulates a list of targets to be compiled into
libMLIR.so.  Note that not all libraries make sense to
be compiled into libMLIR.so.  In particular, we want
to avoid libraries which primarily exist to support
certain tools (such as mlir-opt and mlir-cpu-runner).

Note that the resulting libMLIR.so depends on LLVM, but
does not contain any LLVM components.  As a result, it
is necessary to link with libLLVM.so to avoid linkage
errors. So, libMLIR.so requires LLVM_BUILD_LLVM_DYLIB=on

FYI, Currently it appears that LLVM_LINK_LLVM_DYLIB is broken
because mlir-tblgen is linked against libLLVM.so and
and independent LLVM components.

Previous version of this patch broke depencies on TableGen
targets.  This appears to be because it compiled all
libraries to OBJECT libraries (probably because cmake
is generating different target names).  Avoiding object
libraries results in correct dependencies.

(updated by Stephen Neuendorffer)

Differential Revision: https://reviews.llvm.org/D73130

This reverts commit e17d9c11.
It breaks the build.

Putting this up mainly for discussion on
how this should be done. I am interested in MLIR from
the Julia side and we currently have a strong preference
to dynamically linking against the LLVM shared library,
and would like to have a MLIR shared library.

This patch adds a new cmake function add_mlir_library()
which accumulates a list of targets to be compiled into
libMLIR.so.  Note that not all libraries make sense to
be compiled into libMLIR.so.  In particular, we want
to avoid libraries which primarily exist to support
certain tools (such as mlir-opt and mlir-cpu-runner).

Note that the resulting libMLIR.so depends on LLVM, but
does not contain any LLVM components.  As a result, it
is necessary to link with libLLVM.so to avoid linkage
errors. So, libMLIR.so requires LLVM_BUILD_LLVM_DYLIB=on

FYI, Currently it appears that LLVM_LINK_LLVM_DYLIB is broken
because mlir-tblgen is linked against libLLVM.so and
and independent LLVM components.

Previous version of this patch broke depencies on TableGen
targets.  This appears to be because it compiled all
libraries to OBJECT libraries (probably because cmake
is generating different target names).  Avoiding object
libraries results in correct dependencies.

(updated by Stephen Neuendorffer)

Differential Revision: https://reviews.llvm.org/D73130

This reverts commit 1246e867.

Putting this up mainly for discussion on
how this should be done. I am interested in MLIR from
the Julia side and we currently have a strong preference
to dynamically linking against the LLVM shared library,
and would like to have a MLIR shared library.

This patch adds a new cmake function add_mlir_library()
which accumulates a list of targets to be compiled into
libMLIR.so.  Note that not all libraries make sense to
be compiled into libMLIR.so.  In particular, we want
to avoid libraries which primarily exist to support
certain tools (such as mlir-opt and mlir-cpu-runner).

Note that the resulting libMLIR.so depends on LLVM, but
does not contain any LLVM components.  As a result, it
is necessary to link with libLLVM.so to avoid linkage
errors. So, libMLIR.so requires LLVM_BUILD_LLVM_DYLIB=on

FYI, Currently it appears that LLVM_LINK_LLVM_DYLIB is broken
because mlir-tblgen is linked against libLLVM.so and
and independent LLVM components

(updated by Stephen Neuendorffer)

Differential Revision: https://reviews.llvm.org/D73130

Instead of creating extra libraries we don't really need, collect a
list of all dialects and use that instead.

Differential Revision: https://reviews.llvm.org/D75221

Add an initial version of mlir-vulkan-runner execution driver.
A command line utility that executes a MLIR file on the Vulkan by
translating MLIR GPU module to SPIR-V and host part to LLVM IR before
JIT-compiling and executing the latter.

Differential Revision: https://reviews.llvm.org/D72696

This pass would currently build, but fail to run when this backend isn't
linked in. On the other hand, we'd like it to initialize only the NVPTX
backend, which isn't possible if we continue to build it without the
backend available. Instead of building a broken configuration, let's
skip building the pass entirely.

Differential Revision: https://reviews.llvm.org/D74592

Add basic doxygen config following clang and llvm example with minimal
changes.