Open Ajax Modal Dialog from Controller

Dialogs are often called dialog boxes, modal windows, or pop-up windows. 
Whatever you call them, they are a quick and easy way to display additional information without reloading the entire page. 
Dialog boxes can display just some static text, any node or form of your page, a views page, or any custom markup you'll feed them. 

Drupal core offers different types of dialogs:

•  Modal dialogs
•  Non modal dialogs 
•  Off canvas dialogs

There are many ways to display a dialog box in Drupal, depending on your use case. 
The simplest way to show a dialog box is by simply adding the "use-ajax" class to an HTML anchor tag. 
Dialogs can also be displayed using AJAX Callback Commands like OpenModalDialogCommand or OpenOffCanvasDialogCommand. 

Drupal core offers three different types of dialog boxes that can be displayed without having to reload the entire page:

• Modal dialogs: They overlap the entire page, no other elements can be clicked while modal dialogs are visible. 
  Only one modal popup can be opened at the same time.
• Non modal dialogs: They pop up and stay on top of the page, but still other elements on the page can be clicked. 
  Multiple dialogs can be displayed at the same time.
• Off canvas dialogs: Are no popup windows that overlap other content, but are sliding into the page by moving other content to the side. 
  This type of dialog is especially useful to display larger portions of content, like long detail pages that could require the user to scroll down. 
  Also, the off-canvas dialog is well usable on mobile devices.

When implementing ajax box via the module controller, here are the implementation steps:

step1: creating the controller

create controller with the following header:

    use Drupal\Core\Form\FormBuilder;
    use Drupal\Core\Ajax\AjaxResponse;
    use Drupal\Core\Ajax\OpenModalDialogCommand;   
    use Symfony\Component\HttpFoundation\Request;
    use Symfony\Component\DependencyInjection\ContainerInterface; 

controller must include member: $formBuilder
controller must include methods: __construct, create as follow:

        protected $formBuilder;

        public function __construct(FormBuilder $formBuilder=null) {
            $this->formBuilder = $formBuilder;
        }

        public static function create(ContainerInterface $container) {
            return new static(
                $container->get('form_builder')
            );
        }

create a public method that will activate the modal dialog, using Request as argument is not a must, but will give you ability to use data through the calling mechanism

        public function requestResidenceUpload(Request $request): AjaxResponse {
            $residential = $request->get('residential');
            // Add an AJAX command to open a modal dialog with the form as the content.      
            $response = new AjaxResponse();

            $modal_form = $this->formBuilder->getForm('Drupal\mymodule\Form\MyForm', $residential_id);
                $response->addCommand( new OpenModalDialogCommand('My_Form'), $modal_form, ['width' => 600, 'height'=>600]));
            return $response;        
        } 

as you can see the controller create the ajax box, and activate it with form.
you can create your own form, or use any form.
you can see the the method return AjaxResponse.

Step 2: Creating the form

The custom form must include the following headers

    use Drupal\Core\Form\FormBase;
    use Drupal\Core\Form\FormStateInterface;    
    use Drupal\Core\Ajax\AjaxResponse;
    use Drupal\Core\Ajax\CloseModalDialogCommand;

my form get an extra argument in the following manner:

public function buildForm(array $form, FormStateInterface $form_state, $residential=0) 

with in the buildForm I have created an ajax actions

            $form['actions'] = [
              '#type' => 'actions',
              '#attributes' => [
                'class' => ['form_actions'],
              ],         
            ];
                $form['actions']['cancel'] = [
                  '#type' => 'submit',
                  '#value' => $this->t('Cancel'),
                  '#attributes' => [
                    'class' => ['button','use-ajax','form-submit-modal','action-cancel' ],
                  ],
                  '#ajax' => [
                    'callback' => [$this, 'submitFormClose'],
                    'event' => 'click',
                  ],
                ];       
                $form['actions']['save'] = [
                  '#type' => 'submit',
                  '#value' => $this->t('Save'),
                  '#attributes' => [
                    'class' => ['button','use-ajax','form-submit-modal','action-save' ],
                  ],
                  '#ajax' => [
                    'callback' => [$this, 'submitFormAction'],
                    'event' => 'click',
                    'onclick' => 'javascript:var s=this;setTimeout(function(){s.value="Saving...";s.disabled=true;},1);',
                  ],
                ];

see the both submit actions are defined with class: 'use-ajax'
The form itself should be attached with the following libraries

            $form['#attached']['library'][] = 'core/drupal.dialog.ajax';
            $form['#attached']['library'][] = 'core/drupal.ajax';
            $form['#attached']['library'][] = 'core/jquery.form';  

implementing the action: cancel

        public static function submitFormClose(array &$form, FormStateInterface $form_state) {
            $command = new CloseModalDialogCommand();
            $response = new AjaxResponse();
            $response->addCommand($command);
            return $response;
        }

implementing the action: save

        public static function submitFormUpload(array &$form, FormStateInterface $form_state) {
        	// get the state value
            //$formValue = $form_state->getValue('form_element');
			
			// put here your code
			            
            
			// wrap it with response            
            $response = new AjaxResponse();
            $response->addCommand(new CloseModalDialogCommand()); 
            return $response;
        }        

NOTE: since this callback method is static, all the private methods it is using must be static as well

Step 3: define its routing

since you build a controller, you need to define a routing to its public method in your module.routing.yml

this is it.